vitessio
/
website
mirror of https://github.com/vitessio/website.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Feat: Add a changelog section on hompage (#1907) 

* Feat: Add changelog page to website

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: improve styling and add changelog link on nav

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: add changelog to chinese version

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: update changelog posts with relevant content

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: make changelog section look nicer on mobile

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: fix changelog section to look nicer on mobile

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: add more spacing between texts on mobile

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: nit fixes

Signed-off-by: thisisobate <obasiuche62@gmail.com>

* chore: make section title to be bigger in smaller screens

Signed-off-by: thisisobate <obasiuche62@gmail.com>

---------

Signed-off-by: thisisobate <obasiuche62@gmail.com>
This commit is contained in:
Uchechukwu Obasi 2025-01-10 16:33:03 +01:00 committed by GitHub
parent 22e0c96e57
commit da88655a39
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
19 changed files with 1080 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
assets/sass/style.sass
View File

@ -269,8 +269,20 @@ main.td-main
.mt-100
margin-top: 100px
.changelog > div > .is-size-3
+touch
font-size: 1.4rem !important
.changelog > div > .is-size-4
+touch
font-size: 1.2rem !important
.flex-gap-md
gap: 110px
+tablet
gap: 50px
+touch
gap: 20px
.tag-heading
font-size: 1rem !important
@ -355,6 +367,13 @@ label.date
.flex
display: flex
.flex-1
flex: 1
.flex-col--mobile
+touch
flex-direction: column
.justify-content-space
justify-content: space-between
@ -377,3 +396,17 @@ label.date
.img-thumbnail
height: 222px
object-fit: contain
.changelog
display: flex
flex-direction: column
border-radius: 6px
border: 1px solid #D0D7DF
padding: 48px 40px
+touch
padding: 48px 28px
.text-underlined-on-hover
&:hover
text-decoration: underline

3
config.toml
View File

@ -31,6 +31,9 @@ stackoverflow = "vitess"
[params.blog]
subtitle = "Updates and insights from the **Vitess** team."
[params.changelog]
subtitle = "Track all the changes and updates in the **Vitess** project."
[sitemap]
changefreq = "monthly"
filename = "sitemap.xml"

75
content/en/changelog/2022-12-05-vtadmin-intro.md Normal file
View File

@ -0,0 +1,75 @@
---
author: 'Frances Thai'
date: 2022-12-05
slug: '2022-12-05-vtadmin-intro'
tags: ['Vitess','VTAdmin','vtctld','vtctld2']
title: 'Introducing VTAdmin'
description: "Vitess's cluster management API and UI"
---
[VTAdmin](https://vitess.io/docs/reference/vtadmin/) is now generally available for use! VTAdmin provides both a web client and API for managing multiple Vitess clusters, and is the successor to the now-deprecated UI for [vtctld](https://vitess.io/docs/reference/programs/vtctld/).
## What is VTAdmin?
VTAdmin is made up of two components:
- VTAdmin API: An HTTP(S) and gRPC API server
- VTAdmin Web: A React + Typescript web client built with [Create React App](https://create-react-app.dev/)
## What can you do with VTAdmin?
VTAdmin can do everything the old vtctld2 UI could do, now that the [vtctld2 parity project](https://github.com/vitessio/vitess/projects/13) has been completed. For a complete list of supported API methods, refer [here](https://github.com/vitessio/vitess/blob/main/go/vt/vtadmin/api.go#L332).
The following are just a few examples of what VTAdmin can do!
### VTTablet Management
VTAdmin provides a variety of VTTablet management tools, from starting and stopping replication, setting VTTablets to read/write, reparenting, deleting, pinging, and refreshing VTTablets, to experimental features like VTTablet QPS and VReplication QPS.
<img src="/files/2022-12-05-vtadmin-intro/tablets.gif" alt="GIF of tablets features in VTAdmin Web"/>
_Note: To use experimental features, make sure to set `REACT_APP_ENABLE_EXPERIMENTAL_TABLET_DEBUG_VARS` in VTAdmin Web and `--http-tablet-url-tmpl` in VTAdmin API, as experimental tablet features work by making HTTP requests to the VTTablets._
### Keyspace Management
In VTAdmin, keyspace actions like validating keyspace/schema/version, reloading schemas, rebuilding keyspace graphs and cells, and creating new shards are made easy.
<img src="/files/2022-12-05-vtadmin-intro/keyspaces.gif" alt="GIF of keyspace features in VTAdmin Web"/>
## Workflow Management
VTAdmin allows you to view all your VReplication workflows and monitor workflow streams.
<img src="/files/2022-12-05-vtadmin-intro/workflows.gif" alt="GIF of workflow features in VTAdmin Web"/>
### Topology
The old topology browser in vtctld2 has been reimagined into a graph-traversal UI, which allows you to explore the full topology across single and multi-cluster deployments.
<img src="/files/2022-12-05-vtadmin-intro/topology.gif" alt="GIF of topology in VTAdmin Web"/>
## How does VTAdmin work?
VTAdmin Web is a web client that queries data from VTAdmin API via HTTP protocol. VTAdmin API in turn, is a mostly stateless API server that fetches data from VTGates and Vtctlds via gRPC. It returns this data to the frontend, VTAdmin Web. In a multi-cluster environment, that might look like:
<img src="/files/2022-12-05-vtadmin-intro/architecture.png" alt="Architecture diagram for VTAdmin API and Web"/>
### Lifecycle of a Request
_This is taken from the VTAdmin architecture doc [here](https://vitess.io/docs/reference/vtadmin/architecture/)_.
As an example, take the `/schemas` page in VTAdmin Web:
<img src="/files/2022-12-05-vtadmin-intro/schemas.png" alt="The /schemas page in VTAdmin Web"/>
When a user loads the `/schemas` page in the browser, VTAdmin Web makes an HTTP `GET` `/api/schema/local/commerce/corder` request to VTAdmin API. VTAdmin API then issues gRPC requests to the VTGates and Vtctlds in the cluster to construct the list of schemas. Here's what that looks like in detail:
<img src="/files/2022-12-05-vtadmin-intro/requests.png" alt="Lifecycle of a request to the /schemas page in VTAdmin"/>
## How do I operate VTAdmin?
We have a complete [operator's guide](https://vitess.io/docs/15.0/reference/vtadmin/operators_guide/) to setting up VTAdmin in your Vitess cluster. If you intend to use VTAdmin with the Vitess Operator instead, follow [these instructions](https://vitess.io/docs/15.0/reference/vtadmin/running_with_vtop/).
### Cluster configuration and discovery
VTAdmin API manages to be mostly stateless because it works by proxying requests from clients (VTAdmin Web) to Vitess clusters' VTGates and Vtctlds using gRPC.
The method by which VTAdmin API discovers VTGate and Vtctld addresses to create those gRPC connections is called **cluster discovery**. Users can pass VTGate and Vtctld addresses to VTAdmin API in two ways:
1. As command line arguments at initialization time
2. As an HTTP header cookie or gRPC metadata _after_ initialization time
More information on the different cluster discovery methods, and how to use them, can be found in our [cluster discovery documentation](/docs/15.0/reference/vtadmin//cluster_discovery).
### Role-based access control
VTAdmin also supports role-based access control (RBAC). This allows you to restrict access to, and actions on certain resources to a subset of users for an added layer of security. For more information on how to configure RBAC in VTAdmin, refer to our documentation [here](https://vitess.io/docs/15.0/reference/vtadmin/role-based-access-control/).
## What's next?
There a number of things the team is excited to do next! Some of those things include:
- **Single component VTAdmin**: VTAdmin is currently deployed as two separate components: the Web client and the API server. We're working on packaging these up into a single component much like how the old vtctld2 UI was packaged with Vtctld.
- **Adding VTOrc UI**: We'd also like to combine VTOrc management capabilities into VTAdmin, primarily [the VTOrc UI](https://vitess.io/docs/15.0/user-guides/configuration-basic/vtorc/#old-ui-removal-and-replacement).
- **Adding VTTablet and VTGate features**: VTGate and VTTablet also come with their own web UIs and management APIs - we'd also like to combine these into VTAdmin someday. This includes being able to use the experimental tablet features without providing tablet FQDN templates.
- **Making it easier to deploy**: Since VTAdmin recently went GA, we'd like to work on making the developer experience around deploying VTAdmin much easier. That means adding VTAdmin to existing Makefile workflows and other deployment optimizations.
## Stay in touch with VTAdmin
We welcome you to stay in touch with VTAdmin development in the #feat-vtadmin channel in the Vitess Slack. Here are some other ways you can stay up-to-date:
- **Vitess Docs**: https://vitess.io/docs/15.0/reference/vtadmin/
- **Github Repo**: https://github.com/vitessio/vitess/tree/main/web/vtadmin
- **Github Project**: https://github.com/vitessio/vitess/projects/12

298
content/en/changelog/2023-04-24-schemadiff.md Normal file
View File

@ -0,0 +1,298 @@
---
author: 'Shlomi Noach'
date: 2023-04-24
slug: '2023-04-24-schemadiff'
tags: ['Vitess','MySQL', 'DDL', 'schema']
title: 'schemadiff: Vitess In-memory Schema Diffing, Normalization, Validation and Manipulation'
description: 'Introducing schemadiff, an internal library available in Vitess'
---
Introducing `schemadiff`, an internal library in `Vitess` that has been one of its best-kept secrets until now. At its core, `schemadiff` is a declarative, programmatic library that can produce a diff in SQL format of two entities: tables, views, or full blown database schemas. But it then goes beyond that to normalize, validate, export, and even _apply_ schema changes, all declaratively and without having to use a MySQL server. Let's dive in to understand its functionality and capabilities.
## Some tech specs
`schemadiff` supports MySQL `8.0` dialect and functionality. It is completely in-memory and does not use any MySQL server or MySQL code. It applies MySQL-compatible logic and rules to validate and apply schema changes. It uses a declarative approach but also works with imperative commands.
## Diff objectives
`schemadiff`, as its name suggests, began as a diffing library. The objectives were:
1. Given two table definitions, what schema changes (DDLs) would we need to apply on the first, so it looks like the second?
2. Given two schemas (aka databases), that include tables and views, what schema changes (DDLs) would we need to apply on the first, so it looks like the second?
As a use case for (1), consider [declarative Vitess migrations](https://vitess.io/docs/16.0/user-guides/schema-changes/declarative-migrations/). Consider that you might want to tell your database: "Here's a table, please make it look like so". It's the database's job to determine whether this table exists in the first place or not. If not, then it must be created. If it exists, and already looks exactly like you want it to, great, that's a no-op. But what if the current table looks somewhat different? What changes need to be applied on the existing table so as to make it look like your desired table? Specifically, what `ALTER TABLE` statement(s) (there are some rare scenarios where we might need to invoke more than one) do we need to invoke?
Case (2) is best explained with [PlanetScale branching](https://planetscale.com/docs/concepts/branching). You have a production schema, and you have a local copy of that schema. On your local copy, you make changes over time, and finally wish to apply those changes in production. Did you track those changes? Some ORMs will do a good job at that. But you might not have used an ORM, or the ORM might not be feature complete. What tables do you need to `CREATE`? Which views should be `DROP`ped? And what needs to be `ALTER`ed? Is there a specific order of operations?
There are two approaches to analyzing the diff of tables or schemas. One approach is to use a running MySQL server. Deploy the two schemas on that server, then investigate the `INFORMATION_SCHEMA` metadata tables and construct a model for the schema. `INFORMATION_SCHEMA` offers a formal breakdown such as the columns found in each table, what data type a column has, what default does it have, if any; which indexes are unique, which columns do they cover; what foreign key constraints on a table point to which parent table, what columns are covered; etc.
This approach has two advantages:
1. If you're able to create a table in MySQL, that means it's valid. By the time you introspect `INFORMATION_SCHEMA` on a table, validity is a given. You may safely assume, for example, that keys only cover existing columns.
2. `INFORMATION_SCHEMA` formalizes the majority of information. The data type is well defined. A column precision is an integer value. The text collation is a well known value.
However, there are disadvantages, as well:
- Complex expressions are left, well, complex. A `CREATE VIEW ...` statement is really just a long SQL text. Which tables or views are referenced in our `VIEW` definition? Which columns?
- Not everything is normalized and formalized. We know `int(10)` and `int(11)` are really the same. But as MySQL goes, the column type is different. Contrast this with `varchar(10)` and `varchar(11)`, where the type is truly different. It still takes some higher level logic to understand what is a real diff and what isn't.
- MySQL allows schema inconsistencies. It is possible, in MySQL, to have an "orphaned" `VIEW`, where the tables/views on which it relies, do not exist. Reading something from MySQL doesn't mean it's really valid.
- Last, and most impactful of all, is the operational overhead. To diff two schemas, you need to deploy two schemas, then read back the metadata for all the tables, views, indexes, constraints, etc. You need to deploy a MySQL server, likely in a non-production environment. You will need to start the server, deploy the changes, read back, shutdown the server. This is a heavyweight operation.
And if you then want to validate your _diff_, you probably want to deploy it on the running server, then re-read the result, compare it with what you were expecting to find. This is even more heavyweight.
And, if you want to compute the _order_ in which the changes must take place (some scenarios dictate a specific order, a topic for a future post), then you'd need to solicit confirmation from MySQL by applying changes on the server, iteratively.
The second is the in-memory, programmatic approach. Instead of relying on `INFORMATION_SCHEMA`, we rely on the SQL of the schema, i.e. on the `CREATE TABLE` and `CREATE VIEW` statements themselves. This poses a few challenges of its own:
- First and foremost, you must be able to parse and analyze all statements. This includes a `CREATE TABLE` that has a `CHECK CONSTRAINT` with a complex expression, or a sub-partitioning scheme using functions over columns. You also need the ability to fully parse a `CREATE VIEW` statement, a complex `GENERATED` column expression, etc.
- You must be able to accommodate different equivalent definitions. For example, `create table t (id int primary key)` is equivalent to `create table t(id int, primary key (id))`. Even MySQL's own `SHOW CREATE TABLE` output may present different results depending on when/where the table was created.
- Not having a MySQL server to validate correctness of the initial schema and the generated diffs means that the application must implement that logic.
`schemadiff` uses the programmatic approach. Let's look at some sample code first and then we'll move on to discuss its internals.
## Quick diff examples
By way of simple illustration, we create and diff two schemas, each with a single table. First schema:
```go
schema1, err := NewSchemaFromSQL("create table t (id int, name varchar(64), primary key (id))")
if err == nil {
fmt.Println(schema1.ToSQL())
}
```
```sql
CREATE TABLE `t` (
`id` int,
`name` varchar(64),
PRIMARY KEY (`id`)
);
```
In the second schema, our table is slightly modified:
```go
schema2, err := NewSchemaFromSQL("create table t (id bigint, name varchar(64), key name_idx(name(16)), primary key (id))")
if err == nil {
fmt.Println(schema2.ToSQL())
}
```
```sql
CREATE TABLE `t` (
`id` bigint,
`name` varchar(64),
KEY `name_idx` (`name`(16)),
PRIMARY KEY (`id`)
);
```
We now programmatically diff the two schemas (this is actually the long path to doing so):
```go
hints := &DiffHints{}
diff, err := schema1.SchemaDiff(schema2, hints)
// Handle error
diffs, err := diff.OrderedDiffs()
// Handle error
for _, diff := range diffs {
fmt.Println(diff.CanonicalStatementString())
}
```
And this is the resulting diff:
```sql
ALTER TABLE `t` MODIFY COLUMN `id` bigint, ADD KEY `name_idx` (`name`(16))
```
There are multiple ways of generating the diff. Here's a quick shortcut to achieving the same:
```go
diffs, err := DiffSchemasSQL("create ...", "create ...", hints)
...
```
The main thing to note in the above examples is that everything takes place purely within `go` space, and MySQL is not involved. `schemadiff` is purely programmatic, and makes heavy use of Vitess' [`sqlparser`](https://github.com/vitessio/vitess/tree/main/go/vt/sqlparser) library.
## sqlparser
Vitess is a sharding and management framework running on top of MySQL, which masquerades as a MySQL server to route queries to relevant shards. It thus obviously must be able to parse MySQL's SQL dialect. [`sqlparser`](https://github.com/vitessio/vitess/tree/main/go/vt/sqlparser) is the Vitess library that does so.
`sqlparser` utilizes a classic [yacc](https://en.wikipedia.org/wiki/Yacc) [file](https://github.com/vitessio/vitess/blob/main/go/vt/sqlparser/sql.y) to parse SQL into an Abstract Syntax Tree (AST), with `golang` structs generated and populated by the parser. For example, a SQL `CREATE TABLE` statement is parsed into a [`CreateTable`](https://github.com/vitessio/vitess/blob/ed72037bb6358a2065834589a21f5119fd407136/go/vt/sqlparser/ast.go#L509-L517) instance:
```go
CreateTable struct {
Temp bool
Table TableName
IfNotExists bool
TableSpec *TableSpec
OptLike *OptLike
Comments *ParsedComments
FullyParsed bool
}
```
And here is the breakdown of [`TableSpec`](https://github.com/vitessio/vitess/blob/ed72037bb6358a2065834589a21f5119fd407136/go/vt/sqlparser/ast.go#L1758-L1765):
```go
type TableSpec struct {
Columns []*ColumnDefinition
Indexes []*IndexDefinition
Constraints []*ConstraintDefinition
Options TableOptions
PartitionOption *PartitionOption
}
```
You can already see how AST helps us in analyzing a table's definition. As a very simple illustration, imagine we have two tables we want to diff. Say we want to determine whether the two have a different set of columns (let's ignore ordering for now). How would we do that?
We can programmatically iterate over `range table1.TableSpec.Columns` in each of the tables. We can do a full drill down of all the details in a `ColumnDefinition`. Or, we can take a shortcut. `schemadiff` uses an optimistic approach: most of the schema is likely to be identical. It first attempts to compare components as a whole. If the components are not identical as a whole, we can proceed to drill down.
How do you "compare a component as a whole"? `sqlparser` not only parses SQL, it also exports AST as SQL. We can, for example, run:
```go
s1 := sqlparser.CanonicalString(table1.TableSpec.Columns[i])
s2 := sqlparser.CanonicalString(table2.TableSpec.Columns[i])
```
We can thus compare any two _nodes_. If they export into identical strings, then they are equal and there is no need to drill down. `sqlparser`'s AST also comes with an auto-generated set of deep-equals comparison and deep-copy functionality.
Continuing our simple illustration, if `table2` has a column called `info`, which we cannot find in `table1`, then our table diff, i.e. our `ALTER TABLE` statement, should include an `ADD COLUMN <column definition>` statement. What's the content of the column definition? It is trivially the `CanonicalString()` of the extra column object.
`sqlparser` is devoid of semantic context. It merely deals with a programmatic reflection of SQL. It does not know if a certain table exists, or if a foreign key references valid columns. As long as the syntax is valid, it is satisfied.
## Semantics
The AST's sole purpose is to faithfully represent a SQL query/command. But as a by-product, it can also serve as the base to a semantic analysis of the schema. Consider the following table definition:
```sql
CREATE TABLE `invalid` (
`id` bigint,
`title` varchar(64),
`title` tinytext,
PRIMARY KEY (`val`)
);
```
The above table is _syntactically_ valid, but _semantically_ invalid. There are two columns that go by the same name (`title`), and an index that covers a non-existent column (`val`). These are but two out of many possible errors. A `SHOW CREATE TABLE` would never produce such an output, of course. But we can't assume the output comes from MySQL. If we don't validate the input, and attempt to produce a diff, the results could be unexpected.
## Validation
The basic premise of validation is that all users want their schemas to be valid. There's no point in using invalid schemas, because, at the end of they day, you want to be able to deploy those schemas on real MySQL servers.
To that effect, `schemadiff` enforce validation upon loading a new schema. As we see later on, it also enforces validation upon applying changes. If you try to load an invalid schema as in the above, `schemadiff` returns an informative error. You can't have two columns of same name. Your keys may only cover existing columns. A collation name must be recognized. A `GENERATED` column must refer to existing columns. A `FOREIGN KEY` constraint must reference existing columns in existing tables, and foreign key columns mast have matching data types, etc. Circular `FOREIGN KEY` dependencies are not allowed.
Even more interesting is view validation. `schemadiff` offers a complete validation over view definitions: all referenced tables/views must exist. Circular dependency is not allowed. Referenced columns are validated to exist. Unqualified column names are ensured not to be ambiguous.
## Diff hints
There are nuances. What do you do about different `AUTO_INCREMENT` values? These are not strictly part of the schema, but may or may not cause impact to production if you apply or not apply them. How do you deal with renamed columns? What about constraint names, which, per ANSI SQL, must be unique across the schema? Do you mind if key definitions are in different order?
[`DiffHints`](https://github.com/vitessio/vitess/blob/157857a4c5ee6dcb44e7de08894db8334750746e/go/vt/schemadiff/types.go#L110-L121) includes many controls that affect the diffing logic and output.
## Normalization
Consider this table definition:
```sql
create table what_can_be_normalized (
id int primary key,
i int(12) default null,
v varchar(32) collate utf8mb4_0900_ai_ci
) default charset=utf8mb4 collate=utf8mb4_0900_ai_ci
```
There's a few things to normalize here. Inputs can come in different shapes and sizes, and still mean the same thing. Even MySQL itself presents different output for textual columns based on which version the table was created in. In the above table definition, we can normalize the following:
- The canonical way to declare a `primary key` is as an index definition, not as part of the column definition.
- `int(12)` is just an `int`. `12` does not matter. Integer precision is in fact being [deprecated in MySQL](https://dev.mysql.com/worklog/task/?id=13127). Interestingly, some ORMs do have special treatment for `int(1)`, as an indication to a boolean value. `schemadiff` accommodates that.
- It looks like `i` is `null`able (because it does not say `not null`). Which means its default value is `null` even if we don't explicitly say that.
- `v`'s collation, and thereby character set, agree with the table's collation. It can be removed.
- And, of course, the snippet uses unqualified names and lower case SQL syntax.
A normalized version of the above looks like:
```sql
CREATE TABLE `what_can_be_normalized` (
`id` int,
`i` int,
`v` varchar(32),
PRIMARY KEY (`id`)
) CHARSET utf8mb4,
COLLATE utf8mb4_0900_ai_ci;
```
`schemadiff` chooses to normalize into the shortest form possible, often shorter than MySQL's own `SHOW CREATE TABLE` output. Normalization ensures we converge onto a single, canonical, representation of a table, view, or schema.
## Beyond diff
Up till now, we saw that we can load a schema into `schemadiff`. `schemadiff` would normalize the schema, validate it, and throw an error if it's invalid. We can load a 2nd schema, and if all goes well we can diff the two, and grab the _diffs_.
What then? Do we trust those diffs? Will they result in the correct schema? Will they result in a valid schema?
It looks like to validate those diffs we'd need to run a MySQL server, deploy the 1st schema, apply those diffs (`CREATE`, `ALTER`, `DROP` statements), then re-read the schema and compare it with the one we aimed for. But, this brings back the dependency on a MySQL server, something we wanted to avoid in the first place.
Unless, we can ask `schemadiff` to programmatically _apply_ those diffs and validate the result!
## Applying diffs
Consider this simplified code, or see snippet from [`diff_test.go`](https://github.com/vitessio/vitess/blob/ed72037bb6358a2065834589a21f5119fd407136/go/vt/schemadiff/diff_test.go#L806-L822):
```go
schema1, err := NewSchemaFromSQL(sql1)
// Handle error
schema2, err := NewSchemaFromSQL(sql2)
// Handle error
diff, err := schema1.SchemaDiff(schema2, hints)
// Handle error
diffs, err := diff.OrderedDiffs()
// Handle error
applied, err := schema1.Apply(diffs)
require.NoError(t, err)
appliedDiff, err := schema2.SchemaDiff(applied, hints)
require.NoError(t, err)
assert.True(t, appliedDiff.Empty())
assert.Equal(t, schema2.ToQueries(), applied.ToQueries())
```
`schemadiff` lets us take those _diffs_ and `Apply()` them onto a schema (`schema1`), resulting with a new schema. We expect that result to be equal in content to the expected (`schema2`) schema. We also expect `schemadiff` itself to agree the two are identical.
`Apply()` makes an in-memory schema change. You may `CREATE TABLE` or `ALTER TABLE` or `DROP VIEW`, all programmatically, and all using AST structures such as `CreateTable`, `AlterTable`, or `DropView`, etc.
`Apply()` validates the requested changes on the fly, as well as on the resulting schema. It adheres to MySQL rules. For example:
- If the diff has a `AddColumn` AST struct, verify upfront that no column exists by same name.
- If there's an `AddKey`, validate that the columns specified by the key do in fact exist.
- If there's an `AddKey` and no index name specified, generate one, compatible with MySQL naming.
- Conversely, if there's a `DropColumn`, and some keys exist that actually cover that column, remove the column from those keys; any key that is left without any covered columns is dropped (this is the standard MySQL behavior).
- You shouldn't be able to `DropColumn` if there's a `FOREIGN KEY` referencing that column, though.
- The list goes on.
As we can see, there are a lot of validations involved. Some of them actually depend on each other! Say we drop a column as well as a `FOREIGN KEY` that references that column. MySQL-wise this is a single `ALTER TABLE` operation. But, programmatically, `schemadiff` needs to first remove the `FOREIGN KEY`, and then remove the column. The reverse order is invalid. `schemadiff` computes the correct order of operations for a table.
But then, it also computes the correct order of operations in the grand context of the schema. Consider this diff:
```sql
create view v1 as select * from v2;
create view v2 as select * from t;
```
Alphabetically, `v1` comes before `v2`. But, to apply these two diffs, we must first create `v2`, then `v1`. The reverse order is invalid. This time it's invalid not only programmatically in `schemadiff`, but also when applied to MySQL itself.
`schemadiff` maintains the hierarchical ordering of all tables and views, based on `FOREIGN KEY` and view definitions. There's a specific order for `CREATE` statements, and a specific (reverse) order for `DROP` statements. There may actually be conflicts between the diffs, but that's a topic for a different blog post.
Or, consider the flow in [this `e2e` test](https://github.com/vitessio/vitess/blob/ed72037bb6358a2065834589a21f5119fd407136/go/test/endtoend/schemadiff/vrepl/schemadiff_vrepl_suite_test.go#L345-L428). We hook onto `vitess`'s pre-existing Online DDL tests, a suite that includes a multitude of scenarios. The suite already has a story: a table, a change, an expected result. In our `schemadiff` test we shuffle the logic: we have an original table and and expected table. We evaluate the diff. We apply the diff onto the original table. We expect the result to be identical to the expected table!
Because this runs in GitHub CI, we take advantage of a running MySQL server. We perform the test in-memory, and then also on MySQL itself, and so we have an authoritative validation for the `Apply()` functionality. For fun and glory, we also generate the reverse diff. This test cross validates the programmatic approach, the MySQL schema, and actual MySQL schema changes, and expects full compliance between all.
## Even beyond
The declarative approach is challenging. It requires reverse-engineering of a hidden sequence of changes. Sometimes these changes are inter-dependent. We will discuss this further in a future post.
Note that `schemadiff` is an internal library, and as such, its interface is subject to change. As part of the `Vitess` codebase, `schemadiff` code is open source, licensed under `Apache 2.0`.

104
content/en/changelog/2024-07-22-an-interesting-optimisation.md Normal file
View File

@ -0,0 +1,104 @@
---
author: 'Andrés Taylor'
date: 2024-07-22
slug: '2024-07-22-an-interesting-optimization'
tags: ['Vitess', 'PlanetScale', 'MySQL', 'Query Serving', 'Vindex', 'plan', 'execution plan', 'explain', 'optimizer']
title: 'An Interesting Optimization'
description: "How I implemented an optimization by delaying another optimization"
---
## Introduction
I recently encountered an intriguing bug. A user reported that their query was causing vtgate to fetch a large amount of data, sometimes resulting in an Out Of Memory (OOM) error.
For a deeper understanding of grouping and aggregations on Vitess, I recommend reading this [prior blog post](https://planetscale.com/blog/grouping-and-aggregations-on-vitess).
## The Query
The problematic query was:
```sql
select sum(user.type)
from user
join user_extra on user.team_id = user_extra.id
group by user_extra.id
order by user_extra.id;
```
The planner was unable to delegate aggregation to MySQL, leading to the fetching of a significant amount of data for aggregation.
## Planning and Tree Rewriting
During the planning phase, we perform extensive tree rewriting to push as much work down under Routes as possible. This involves repeatedly rewriting the tree until no further changes occur during a full pass of the tree, a state known as fixed-point. The goal of this rewriting process is to optimize query execution by pushing operations closer to the data.
### Initial Plan
The first plan after horizon expansion looked like this:
```
Ordering (user_extra.id)
└── Aggregator (ORG sum(`user`.type), user_extra.id group by user_extra.id)
└── ApplyJoin on [`user`.team_id | :user_team_id = user_extra.id | `user`.team_id = user_extra.id]
├── Route (Scatter on user)
│ └── Table (user.user)
└── Route (Scatter on user)
└── Filter (:user_team_id = user_extra.id)
└── Table (user.user_extra)
```
## Trying to Optimize the Plan
We don't split aggregation between MySQL and vtgate in the initial phases, so we couldn't immediately push down the aggregation through the join. However, we can push down ordering under the aggregation.
### Pushing Ordering Under Aggregation
By pushing ordering under aggregation, the plan changes to:
```
Aggregator (ORG sum(`user`.type), user_extra.id group by user_extra.id)
└── Ordering (user_extra.id)
└── ApplyJoin on `user`.team_id = user_extra.id
...
```
We can't push the ordering further down since it's sorted by the right hand side of the join. Ordering can only be pushed down to the left hand side.
This leaves us in an unfortunate situation - ordering is blocking the aggregator from being pushed down, which means we have to fetch all that data, _and_ sort it to do the aggregation.
### The Solution
The solution I typically use in these situations involves leveraging the phases we have in the planner.
### Phases
We have several phases that run sequentially. After completing a phase, we run the push-down rewriters, then move to the next phase, and so on.
Rewriters perform one of two functions:
1. Running a rewriter over the plan to perform a specific task. For example, the "pull DISTINCT from UNION" rewriter extracts the DISTINCT part from UNION and uses a separate operator for it.
2. Controlling when push-down rewriters are enabled. Some rewriters only turn on after reaching a certain phase.
By delaying the "ordering under aggregation" rewriter until the "split aggregation" phase, we can push down the aggregation under the join. This doesn't stop the "ordering under aggregation" rewriter from doing its job, it just has to wait a bit before doing it.
The final tree looks like this:
```
Aggregator (sum(`user`.type) group by user_extra.col)
└── Projection (sum(`user`.type) * count(*), user_extra.col)
└── Ordering (user_extra.col)
└── ApplyJoin (on [`user`.team_id = user_extra.id])
├── Route (Scatter on user)
│ └── Aggregator (sum(type) group by team_id)
│ └── Table (user)
└── Route (Scatter on user_extra)
└── Aggregator (count(*) group by user_extra.col)
└── Filter (:user_team_id = user_extra.id)
└── Table (user_extra)
```
Most of the aggregation has been pushed down to MySQL, and at the vtgate level, we are left with only SUMming the SUMs we get from each shard.
## Conclusion
This optimization demonstrates the complexity of query planning and the importance of efficient tree rewriting in Vitess. By carefully pushing operations closer to the data, we can significantly improve query performance and resource utilization.
For more details on the implementation, you can check out the [pull request on GitHub](https://github.com/vitessio/vitess/pull/16278) that addresses this optimization.

20
content/en/changelog/2024-08-23-recursive-cte.md Normal file
View File

@ -0,0 +1,20 @@
---
author: 'Andrés Taylor'
date: 2024-08-23
slug: '2024-08-23-recursive-cte'
tags: ['Vitess', 'PlanetScale', 'MySQL', "Compatibility", "CTE"]
title: 'Vitess Now Supports Recursive CTEs: A Step Closer to Full MySQL Compatibility'
description: "Vitess introduces support for recursive CTEs, enabling powerful query capabilities across sharded keyspaces, as we continue our progress toward full MySQL feature compatibility"
---
We are excited to announce that Vitess now supports recursive [Common Table Expressions (CTEs)](https://dev.mysql.com/doc/refman/8.4/en/with.html), marking another significant step in our journey to fully align with MySQL’s capabilities. Recursive CTEs, often a critical feature for complex query handling, allow for the execution of recursive queries within a single CTE. This addition brings more flexibility and power to developers using Vitess, especially those working with distributed databases.
One of the key challenges in implementing recursive CTEs within a sharded environment is managing the distribution of data across multiple shards. Vitess has addressed this challenge with two distinct approaches. First, when possible, we merge recursive CTEs into a single query that can be efficiently executed on a single shard. This optimization makes it possible to run recursive queries on a single shard, for queries where this is possible.
In scenarios where merging is not feasible, Vitess takes advantage of its powerful `vtgate` proxy. The `vtgate` handles recursion, allowing recursive CTEs to function seamlessly across sharded keyspaces. This ensures that recursive queries are no longer a barrier when working with large, distributed datasets.
It’s important to note that support for recursive CTEs is still in the experimental stage and has just been merged into the main branch. This feature is not yet available in any official release but will be part of the upcoming Vitess 21 release. We encourage the community to explore this feature and provide feedback on any issues encountered. Your input is invaluable as we continue to refine and enhance Vitess.
This development brings us even closer to our goal of fully supporting MySQL’s feature set. With recursive CTEs now implemented, Vitess is on the verge of achieving complete MySQL compatibility. We remain dedicated to expanding Vitess’s capabilities, and this advancement marks another significant milestone in that ongoing journey.
We look forward to your feedback and hope you enjoy the expanded capabilities that recursive CTEs bring to Vitess.

4
content/en/changelog/_index.md Normal file
View File

@ -0,0 +1,4 @@
---
title: The Vitess changelog
aliases: ['/zh/changelog/']
---

108
content/zh/changelog/2024-03-06-announcing-vitess-19.md Normal file
View File

@ -0,0 +1,108 @@
---
author: 'Vitess Maintainer Team'
date: 2024-03-06
slug: '2024-03-06-announcing-vitess-19'
tags: ['release','Vitess','MySQL','kubernetes','operator','sharding', 'Orchestration', 'Failover', 'High-Availability']
title: 'Announcing Vitess 19'
description: "Vitess 19 is now Generally Available"
---
## Announcing Vitess 19
We're thrilled to announce the release of Vitess 19, our latest version packed with enhancements aimed at improving scalability, performance, and usability of your database systems. With this release, we continue our commitment to providing a powerful, scalable, and reliable database clustering solution for MySQL.
## What's New in Vitess 19
* **Dropping Support for MySQL 5.7**: As Oracle has marked MySQL 5.7 end of life in October 2023, we're also moving forward by dropping support for MySQL 5.7. We advise users to upgrade to MySQL 8.0 while on Vitess 18 before making the jump to Vitess 19. However, Vitess 19 will still support importing from MySQL 5.7.
* **Deprecations**: We're cleaning house to streamline our offerings and improve maintainability. This includes deprecating several VTTablet flags, mysql specific tags of the Docker image `vitess/lite`, and changes to the `EXPLAIN` statement format.
* **Breaking Changes**: Notably, `ExecuteFetchAsDBA` now rejects multi-statement SQL, enforcing stricter security and stability practices.
* **New Metrics**: We're introducing new metrics for stream consolidations and adding the build version to `/debug/vars` to provide deeper insights and traceability.
* **Enhanced Query Compatibility**: This release brings support for multi-table delete operations, a new `SHOW VSCHEMA KEYSPACES` query, and several other SQL syntax enhancements that broaden Vitess's compatibility with MySQL.
* **Apply VSchema Enhancements**: We've added a `--strict` sub-flag and corresponding gRPC field to the `ApplyVSchema` command, ensuring that only known parameters are used in Vindexes, enhancing error checking and config validation.
* **Tablet Throttler**: Throttlers now communicate via gRPC only. HTTP communication is no longer used. This closes a possible vulnerability vector.
* **Online DDL**: Support backoff for cut-over attempts in face of locking. Support forced cut-over.
* **Incremental Backup**: Support backup names and empty backups.
* **Table lifecycle**: Quicker cleanup flow.
* **Performance improvements**: Including a new connection pool for the Tablets, faster hashing in sharded Vitess clusters and faster aggregations in the Gates.
## Dive Deeper
Let's take a closer look at some of the key features.
### Query Compatibility Enhancements
Vitess 19 introduces several SQL syntax improvements and compatibility features, including:
* Support for `AVG()` aggregation function on sharded keyspaces, utilizing a combination of `SUM` and `COUNT`.
* Non-recursive Common Table Expressions (CTEs) support, allowing for more complex query constructions.
### Tablet throttler
Inter-throttler communication is now solely based on gRPC. HTTP communication is no longer supported.
### Online DDL
Vitess migration cut-over now uses back-off in face of table locks. If unable to cut-over, next attempts take place in increasing intervals. This reduces the impact on an already overloaded production system.
Online DDL also supports forced cut-over, at either predetermined timeout or on demand. Forced cut-over prioritizes the cut-over completion over production traffic, and terminates queries and transactions that conflict with the cut-over.
See [https://github.com/vitessio/vitess/pull/14546](https://github.com/vitessio/vitess/pull/14546).
### Incremental backup
The flag `Backup|BackupShard –incremental-from-pos` accepts a backup name as the backup starting point.
An empty incremental backup is now allowed, and the `Backup|BackupShard` command returns with success error code, although no backup manifest or other artifacts are created.
### Table Lifecycle
The table GC mechanism is now more responsive to tables that need to be garbage collected, and is able to observe operations that generate GC tables. For example, it can capture the result of an `ALTER VITESS_MIGRATION … CLEANUP` command and move the table through the relevant stages within seconds rather than taking several minutes or hours.
### Breaking Change: `ExecuteFetchAsDBA`
The command `ExecuteFetchAsDBA` now rejects multi-statement input. Previously, the results of multi-statement input were implicitly allowed, but resulted in undefined and undesired behavior: errors were only reported for the first statement, and silently dropped for successive statements. The connection was left in an undefined state and could leak results to next users of the connection pool. Schema tracker would not be notified of changes until the connection was closed. We will introduce formal multi-statement support in a future version.
### Performance Improvements
Following the trend over the past 3 years, this new Vitess release is faster than the previous one in _all_ the benchmarks we track in [Arewefastyet](https://benchmark.vitess.io/status). We've fixed several performance regressions from Vitess 18 and introduced significant performance improvements.
#### New connection pool
The connection pool for MySQL connections in the Tablets has been rewritten from scratch. The new pool is architected over several lock-free stacks and provides significantly lower query latencies, lower and more fair wait times and more efficient usage for idle connections. This is particularly noticeable in Vitess clusters with external tablets (i.e. clusters where the Tablet and the MySQL instance are deployed in different hosts) and busy Vitess clusters with many point queries.
#### Faster hashing in sharded Vitess clusters
The VIndex hasher for textual columns was previously implemented using the `x/text/collate` package, which allocates a linear amount of memory based on the length of the column being hashed. We've replaced it with a custom, backwards-compatible implementation, which is both faster and uses a constant amount of memory. This is a very significant performance improvement for sharded tables that use a large textual columns as a sharding key.
#### Faster comparisons in cross-shard aggregations
The performance of cross-shard aggregations that use `ORDER` or `GROUP BY` qualifiers has been greatly improved by introducing Tiny Weights. The query executor in the VTGates now tags all the SQL values from the upstream shards with a compressed form of their weight string, allowing constant-time comparisons while performing aggregations.
### A Call to the Community
We're excited to see how you'll use Vitess 19 to scale your database systems. As always, we're eager to hear your feedback and experiences. Join us on our [GitHub](https://github.com/vitessio/vitess) or [Slack channel](http://vitess.io/slack) to share your stories, ask questions, and connect with the Vitess community.
### Getting Started
Upgrading to Vitess 19 is straightforward, but we recommend reviewing the [detailed release notes](https://github.com/vitessio/vitess/blob/main/changelog/19.0/19.0.0/release_notes.md) for a smooth transition. Check out our [documentation](https://vitess.io/docs/) for comprehensive guides and tips.
Thank you for your continued support and contributions to the Vitess project. Here's to making database scaling even easier and more efficient with Vitess 19!
---
_The Vitess Team_

95
content/zh/changelog/2024-06-27-announcing-vitess-20.md Normal file
View File

@ -0,0 +1,95 @@
---
author: 'Vitess Maintainer Team'
date: 2024-06-27
slug: '2024-06-27-announcing-vitess-20'
tags: ['release', 'Vitess', 'v20', 'MySQL', 'kubernetes', 'operator', 'vreplication', 'multi-tenancy', 'Usability', 'Online DDL']
title: 'Announcing Vitess 20'
description: "Vitess 20 is now Generally Available"
---
We're delighted to announce the release of [Vitess 20](https://github.com/vitessio/vitess/releases/tag/v20.0.0) along with [version 2.13.0](https://github.com/planetscale/vitess-operator/releases/tag/v2.13.0) of the Vitess Kubernetes Operator.
Version 20 focuses on usability and maturity of existing features, and continues to build on the solid foundation of scalability and performance established in previous versions. Our commitment remains steadfast in providing a powerful, scalable, and reliable solution for your database scaling needs.
## What's New in Vitess 20
- **Query Compatibility**: enhanced DML support including improved query compatibility, Vindex hints, and extended support for various sharded `update` and `delete` operations.
- **VReplication**: multi-tenant imports (experimental).
- **Online DDL**: improved support for various schema change scenarios, dropping support for `gh-ost`.
- **Vitess Operator**: automated and scheduled backups.
## Dive Deeper
Let’s look into some key highlights of this release.
### Query Compatibility
The latest Vitess release enhances DML support with features like Vindex hints, sharded updates with limits, multi-table updates, and advanced delete operations.
Vindex hints enable users to influence shard routing:
```sql
SELECT * FROM user USE VINDEX (hash_user_id, secondary_vindex) WHERE user_id = 123;
SELECT * FROM order IGNORE VINDEX (range_order_id) WHERE order_date = '2021-01-01';
```
Sharded updates with limits are now supported:
```sql
UPDATE t1 SET t1.foo = 'abc', t1.bar = 23 WHERE t1.baz > 5 LIMIT 1;
```
Multi-table updates and multi-target updates enhance flexibility:
```sql
UPDATE t1 JOIN t2 ON t1.id = t2.id JOIN t3 ON t1.col = t3.col SET t1.baz = 'abc', t1.apa = 23 WHERE t3.foo = 5 AND t2.bar = 7;
UPDATE t1 JOIN t2 ON t1.id = t2.id SET t1.foo = 'abc', t2.bar = 23;
```
Advanced delete operations with subqueries and multi-target support are included:
```sql
DELETE FROM t1 WHERE id IN (SELECT col FROM t2 WHERE foo = 32 AND bar = 43);
DELETE t1, t3 FROM t1 JOIN t2 ON t1.id = t2.id JOIN t3 ON t1.col = t3.col;
```
These features provide greater control and efficiency for managing sharded data. For more details, please refer to the Vitess and MySQL documentation.
### VReplication: Multi-tenant Imports (experimental)
Many web-scale applications use a multi-tenant architecture where each tenant has their own database (with identical schemas). There are several challenges with this approach like provisioning and scaling potentially tens of thousands of databases, and uniformly updating database schemas across them.
A sharded Vitess [keyspace](https://vitess.io/docs/concepts/keyspace/) is a great option for such a system with a single logical database serving all tenants. Vitess 20 adds support for importing data from such a multi-tenant setup into a single Vitess [keyspace](https://vitess.io/docs/concepts/keyspace/), with new [`--shards` and `--tenant-id` flags](https://vitess.io/docs/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_create/) for the [MoveTables workflow](https://vitess.io/docs/reference/vreplication/movetables/). You would run one such workflow for each tenant, with imported tenants being served by the Vitess cluster.
### Online DDL
Vitess migrations now support `enum` definition reordering. Vitess opts to use `enum`s by alias (their string representation) rather than by ordinal value (the internal integer representation).
Vitess now has better analysis for `INSTANT` DDL scenarios, enabled with the `--prefer-instant-ddl` DDL [strategy flag](https://vitess.io/docs/20.0/user-guides/schema-changes/ddl-strategy-flags/). It is able to predict whether a migration can be fulfilled by the `INSTANT` algorithm and use this algorithm if so.
It also improves support for range partitioning migrations, and opts to use direct partitioning queries over Online DDL where appropriate.
VDiffs can now be run on Online DDL Workflows which are still in progress (i.e. not yet cut-over).
Release 20.0 drops support for `gh-ost` for Online DDL, as we continue to invest in `vitess` migrations based on VReplication. The `gh-ost` strategy is still recognized; however:
- Vttablet binaries no longer bundle the `gh-ost` binary. The user should provide their own `gh-ost` binary, and supply `vttablet --gh-ost-path`.
- Vitess no longer tests `gh-ost` in CI/endtoend tests.
### Vitess-operator
Automated and scheduled backups are now available as an experimental feature in v2.13.0. We have added a [new user guide](https://vitess.io/docs/20.0/user-guides/operating-vitess/backup-and-restore/scheduled-backups/) for this feature.
## Vitess and the Community
As an open-source project, Vitess thrives on the contributions, insights, and feedback from the community. Your experiences and input are invaluable in shaping the future of Vitess. We encourage you to share your stories and ask questions, on [GitHub](https://github.com/vitessio/vitess) or in our [Slack community](http://vitess.io/slack).
## Getting Started
For a seamless transition to [Vitess 20](https://github.com/vitessio/vitess/releases/tag/v20.0.0), we highly recommend reviewing the [detailed release notes](https://github.com/vitessio/vitess/blob/main/changelog/20.0/20.0.0/release_notes.md). Additionally, you can explore [our documentation](https://vitess.io/docs/20.0/) for guides, best practices, and tips to make the most of Vitess 20. Whether you're upgrading from a previous version or running Vitess for the first time, our resources are designed to support you every step of the way.
Thank you for your support and contributions to the Vitess project!
---
_The Vitess Team_

20
content/zh/changelog/2024-08-23-recursive-cte.md Normal file
View File

@ -0,0 +1,20 @@
---
author: 'Andrés Taylor'
date: 2024-08-23
slug: '2024-08-23-recursive-cte'
tags: ['Vitess', 'PlanetScale', 'MySQL', "Compatibility", "CTE"]
title: 'Vitess Now Supports Recursive CTEs: A Step Closer to Full MySQL Compatibility'
description: "Vitess introduces support for recursive CTEs, enabling powerful query capabilities across sharded keyspaces, as we continue our progress toward full MySQL feature compatibility"
---
We are excited to announce that Vitess now supports recursive [Common Table Expressions (CTEs)](https://dev.mysql.com/doc/refman/8.4/en/with.html), marking another significant step in our journey to fully align with MySQL’s capabilities. Recursive CTEs, often a critical feature for complex query handling, allow for the execution of recursive queries within a single CTE. This addition brings more flexibility and power to developers using Vitess, especially those working with distributed databases.
One of the key challenges in implementing recursive CTEs within a sharded environment is managing the distribution of data across multiple shards. Vitess has addressed this challenge with two distinct approaches. First, when possible, we merge recursive CTEs into a single query that can be efficiently executed on a single shard. This optimization makes it possible to run recursive queries on a single shard, for queries where this is possible.
In scenarios where merging is not feasible, Vitess takes advantage of its powerful `vtgate` proxy. The `vtgate` handles recursion, allowing recursive CTEs to function seamlessly across sharded keyspaces. This ensures that recursive queries are no longer a barrier when working with large, distributed datasets.
It’s important to note that support for recursive CTEs is still in the experimental stage and has just been merged into the main branch. This feature is not yet available in any official release but will be part of the upcoming Vitess 21 release. We encourage the community to explore this feature and provide feedback on any issues encountered. Your input is invaluable as we continue to refine and enhance Vitess.
This development brings us even closer to our goal of fully supporting MySQL’s feature set. With recursive CTEs now implemented, Vitess is on the verge of achieving complete MySQL compatibility. We remain dedicated to expanding Vitess’s capabilities, and this advancement marks another significant milestone in that ongoing journey.
We look forward to your feedback and hope you enjoy the expanded capabilities that recursive CTEs bring to Vitess.

176
content/zh/changelog/2024-10-29-announcing-vitess-21.md Normal file
View File

@ -0,0 +1,176 @@
---
author: 'Vitess Maintainer Team'
date: 2024-10-29
slug: '2024-10-29-announcing-vitess-21'
tags: [ 'release', 'Vitess', 'v21', 'MySQL', 'kubernetes', 'operator', 'vreplication', 'multi-tenancy', 'Usability',
'Online DDL' ]
title: 'Announcing Vitess 21'
description: "Vitess 21 is now Generally Available"
---
# **Announcing Vitess 21**
We're delighted to announce the release of [Vitess 21](https://github.com/vitessio/vitess/releases/tag/v21.0.0) along
with [version 2.14.0](https://github.com/planetscale/vitess-operator/releases/tag/v2.14.0) of the Vitess Kubernetes
Operator.
Version 21 focuses on enhancing query compatibility, improving cluster management, and expanding VReplication
capabilities, with experimental support for atomic distributed transactions and recursive CTEs. Key features include
reference table materialization, multi-metric throttler support, and enhanced Online DDL functionality. Backup and
restore processes benefit from a new **mysqlshell** engine, while **vexplain** now offers detailed execution traces and
schema analysis. The Vitess Kubernetes Operator introduces horizontal auto-scaling for VTGate pods and Kubernetes 1.31
support, improving overall scalability and deployment flexibility.
## What's New in Vitess 21
* **Query Compatibility**: Experimental support for atomic distributed transactions and recursive CTEs
* **VReplication**: Reference table materialization, dynamic workflow configuration
* **Cluster Management And VTOrc**: More metrics in VTOrc to track errant GTIDs
* **Throttler**: multi-metric support
* **Online DDL**: Various improvements
* **Backup & restore**: Experimental mysqlshell engine
* **Vitess Operator**: VTGate scaling, image customization, Kubernetes 1.31 support
* **VTAdmin**: VReplication workflow creation and management, distributed transaction management
* **VExplain**: `vexplain trace` for detailed query execution insights, `vexplain keys` for analyzing sharding key
usage and optimizing query performance
## Let’s Dive Deeper
Let’s take a deeper look at some key highlights of this release.
### Query Compatibility
#### Atomic Distributed Transactions
We’re reintroducing atomic distributed transactions with a revamped, more resilient design. This feature now offers
deeper integration with core Vitess components and workflows, such as Online DDL and VReplication (including operations
like **MoveTables** and **Reshard**). We have also greatly simplified the configuration required to use atomic
distributed transactions. This feature is currently in an experimental state, and we encourage you to explore it and
share your feedback to help us improve it further.
#### Recursive Common Table Expressions (CTEs)
Vitess 21 introduces experimental support for recursive CTEs, allowing more complex hierarchical queries and graph
traversals. This feature enhances query flexibility, particularly for managing parent-child relationships like
organizational structures or tree-like data. As this functionality is still experimental, we encourage you to explore it
and provide feedback to help us improve it further.
### Cluster Management and VTOrc
We have added a new metric in VTOrc that shows
the count of errant GTIDs in all the tablets for better visibility and alerting. This will help
operators to track and manage errant GTIDs across the cluster.
### VReplication
#### Reference Table Materialization
Vitess provides **Reference Tables** as a mechanism to replicate commonly used lookup tables from an unsharded keyspace
into all shards in a sharded keyspace. Such tables might be used to hold lists of countries, states, zip codes, etc,
which are commonly used in joins with other tables in the sharded keyspace. Using reference tables allows Vitess to
execute joins in parallel on each shard thus avoiding cross-shard joins. Previously, we recommended creating Materialize
workflows for reference tables, but did not provide an easy way to do so. In v21 we have added explicit support to the
**Materialize** command to replicate a set of reference tables into a sharded keyspace.
#### Dynamic Workflow Configuration
Previously, many configuration options for VReplication workflows were controlled by VTTablet flags. This meant that any
change required restarting all VTTablets. We now allow these to be overridden while creating a workflow or updated
dynamically once the workflow is in progress.
### Throttler: multi-metric support
The tablet throttler has been redesigned
with [new multi-metric support](https://vitess.io/docs/21.0/reference/features/tablet-throttler/). With this, the
throttler now handles more than just replication lag or custom queries, but instead can work with multiple metrics at
the same time, and check for different metrics for different clients or for different workflows. This gives users better
control over the throttler allowing them to fine-tune its behavior based on their specific production requirements.
Several new metrics have been introduced in v21, with plans to expand the list of available metrics in later versions.
The multi-metric throttler in v21 is backward compatible with the v20 throttler. It is possible to have a v20 primary
tablet collecting throttler data from a v21 replica tablet, and vice versa. This backward compatibility will be removed
in v22, where all tablet throttlers will be expected to communicate multi-metric data.
Other key throttler changes:
* With the above, the sub-flags `--check-as-check-self` and `--check-as-check-shard` to
the `UpdateThrottlerConfig` command are deprecated and slated to be removed in a future version.
* Similarly, `SHOW VITESS_THROTTLER STATUS` and `SHOW VITESS_THROTTLED_APPS` queries, and all `/throttler/`
API access points (with the exception of `/throttler/check`) are deprecated and slated to be removed in v22.
* When enabled, the throttler ensures it leases heartbeat updates, even if heartbeat configuration is otherwise unset.
In other words, the throttler overrides the configuration when it requires heartbeat information.
* Throttler check response now includes a human readable summary detailing exactly why a request was rejected (if
rejected).
### Online DDL
Several bug fixes and improvements, including:
* Added support for the `ALTER VITESS_MIGRATION CLEANUP ALL` command.
* More `INSTANT` DDL scenario analysis, going further beyond the documented limitations.
* In schema changes where columns change charsets, Online DDL now converts the text programmatically rather than using
a `CONVERT(... USING utf8mb4)` clause, thereby improving performance when such columns are part of the Primary Key
or the iteration key.
* Internally, more of the schema and diff analysis is now delegated to `schemadiff` library, which means more
programmatic power and better testability.
* Fixes for self-referencing foreign key tables (only relevant when using the PlanetScale MySQL build).
### Backup & restore
Introducing an experimental [mysqlshell engine](https://vitess.io/docs/21.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup/#using-mysqlshell-experimental). With this engine it is possible to run logical backups and restores. The mysqlshell engine can be used to create
full backups, incremental backups and point in time recoveries. It is also available to use with the Vitess Kubernetes
Operator.
The **mysqlshell** engine work was contributed by the Slack engineering team.
### VExplain Enhancements
#### VExplain Trace
The new **vexplain trace** command provides deeper insights into query execution paths by capturing
detailed execution traces. This helps developers and DBAs analyze performance bottlenecks, review query plans, and gain
visibility into how Vitess processes queries across distributed nodes. The trace output is delivered as a JSON object,
making it easy to integrate with external analysis tools.
#### VExplain Keys
The new **vexplain keys** feature helps you analyze how your queries interact with your schema,
showing which columns are used in filters, groupings, and joins across tables. This tool is especially useful for
identifying candidate columns for indexing, sharding, or optimization, whether you’re using Vitess or a standalone MySQL
setup. By providing a clear view of column usage, **vexplain keys** makes it easier to
fine-tune your database for better performance, regardless of your backend infrastructure.
### Vitess Kubernetes Operator
Vitess v21.0.0 comes with a companion release of
the [vitess-operator v2.14.0](https://github.com/planetscale/vitess-operator/releases/tag/v2.14.0). In v2.14 we have
added the ability to horizontally scale the VTGate deployment using an HPA. We have upgraded the supported version of
Kubernetes to the latest version (v1.31). We have added a feature that allows users to select Docker images on a
per-keyspace basis instead of a single setting for the entire cluster.
### VTAdmin
New VTAdmin pages have been added for creating, monitoring and managing VReplication Workflows. We have also added a
dashboard to view and conclude distributed transactions.
## Vitess and the Community
As an open-source project, Vitess thrives on the contributions, insights, and feedback from the community. Your
experiences and input are invaluable in shaping the future of Vitess. We encourage you to share your stories and ask
questions, on [GitHub](https://github.com/vitessio/vitess) or in our [Slack community](http://vitess.io/slack).
## Getting Started
For a seamless transition to [Vitess 21](https://github.com/vitessio/vitess/releases/tag/v21.0.0), we highly recommend
reviewing the [detailed release notes](https://github.com/vitessio/vitess/tree/main/changelog/21.0). Additionally, you
can explore our documentation for guides, best practices, and tips to make the most of Vitess 21. Whether you're
upgrading from a previous version or running Vitess for the first time, our resources are designed to support you every
step of the way.
Thank you for your support and contributions to the Vitess project!
---
*The Vitess Maintainer Team*

4
content/zh/changelog/_index.md Normal file
View File

@ -0,0 +1,4 @@
---
title: The Vitess changelog
aliases: ['/en/changelog/']
---

3
i18n/en.toml
View File

@ -40,6 +40,9 @@ other = "Docs"
[blog]
other = "Blog"
[changelog]
other = "Changelog"
[community]
other = "Community"

3
i18n/zh.toml
View File

@ -40,6 +40,9 @@ other = "文档"
[blog]
other = "博客"
[changelog]
other = "更新日志"
[community]
other = "社区"

61
layouts/changelog/section.html Normal file
View File

@ -0,0 +1,61 @@
{{ define "main" }}
{{ $subtitle := site.Params.changelog.subtitle | markdownify }}
{{ $posts := where site.RegularPages "Section" "changelog" }}
{{ partial "navbar.html" . }}
{{ partial "blog/hero.html" . }}
<section class="section">
<div class="container">
<div class="columns">
<div class="column is-3 has-text-right">
<p class="title is-size-1 has-text-weight-light">
{{ .Title }}
</p>
<hr class="has-background-primary" />
<p class="is-size-4 has-text-weight-light">
{{ $subtitle }}
</p>
</div>
<div class="column is-8 is-blog-container">
{{ range $posts }}
{{ $date := dateFormat "January 2, 2006" .Date }}
{{ $summary := .Summary | truncate 250 | markdownify }}
<div class="card is-blog-card">
<div class="card-content">
<div class="media">
<div class="media-content">
<a class="title is-size-3 has-text-weight-bolder is-blog-post-title" href="{{ .RelPermalink }}">
{{ .Title }}
</a>
<nav class="level is-size-4">
<div class="level-left">
<span class="has-text-weight-bolder">
{{ .Params.author }}
</span>
</div>
<div class="level-right">
<span class="has-text-weight-light">
{{ $date }}
</span>
</div>
</nav>
<hr />
<div class="content is-blog-summary">
{{ $summary }}
</div>
</div>
</div>
</div>
</div>
{{ end }}
</div>
</div>
</div>
</section>
{{ end }}

44
layouts/changelog/single.html Normal file
View File

@ -0,0 +1,44 @@
{{ define "main" }}
{{ $date := dateFormat "January 2, 2006" .Date }}
{{ partial "navbar.html" . }}
{{ partial "blog/hero.html" . }}
<section class="section">
<div class="container">
<div class="columns">
<div class="column is-blog-container">
<div class="card is-blog-card">
<div class="card-content">
<div class="media">
<div class="media-content">
<a class="title is-size-3 has-text-weight-bolder is-blog-post-title" href="{{ .RelPermalink }}">
{{ .Title }}
</a>
<nav class="level is-size-4">
<div class="level-left">
<span class="has-text-weight-bolder">
{{ .Params.author }}
</span>
</div>
<div class="level-right">
<span class="has-text-weight-light">
{{ $date }}
</span>
</div>
</nav>
<hr />
<div class="content">
{{ .Content }}
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
{{ end }}

1
layouts/index.html
View File

@ -5,4 +5,5 @@
{{ partial "home/users.html" . }}
{{ partial "home/video.html" . }}
{{ partial "home/cncf.html" . }}
{{ partial "home/changelog.html" . }}
{{ end }}

24
layouts/partials/home/changelog.html Normal file
View File

@ -0,0 +1,24 @@
<section class="section is-light mt-40">
<div class="changelog container">
<div class="flex justify-content-space">
<h2 class="title is-size-3 has-text-weight-bolder">Changelog</h2>
<a href="/changelog/">view all changes</a>
</div>
<div class="flex flex-col--mobile flex-gap-md">
{{- $changelogList := where .Site.RegularPages "Section" "changelog" -}} {{-
range $index, $changelog := $changelogList | first 4 -}}
<div class="flex-1">
<a
class="title is-3 is-size-4 has-text-weight-bolder text-underlined-on-hover"
href="{{ .RelPermalink }}"
>
{{ $changelog.Title }}
</a>
<div class="has-text-weight-light">
{{ $changelog.Date.Format "January 2, 2006" }}
</div>
</div>
{{- end -}}
</div>
</div>
</section>

4
layouts/partials/navbar.html
View File

@ -34,6 +34,10 @@
{{ T "blog" }}
</a>
<a class="navbar-item has-text-weight-bold" href="{{ "/changelog" | relLangURL }}">
{{ T "changelog" }}
</a>
<a class="navbar-item has-text-weight-bold" href="{{ "/community" | relLangURL }}">
{{ T "community" }}
</a>