ent/doc/md/versioned-migrations.md

id, title

id	title
versioned-migrations	Versioned Migrations

If you are using the Atlas migration engine you are able to use the versioned migrations feature of it. Instead of applying the computed changes directly to the database, it will generate a set of migration files containing the necessary SQL statements to migrate the database. These files can then be edited to your needs and be applied by any tool you like (like golang-migrate, Flyway, liquibase).

Generating Versioned Migration Files

As mentioned above, versioned migrations do only work, if the new Atlas based migration engine is used. Migration files are generated by computing the difference between two states. We call the state reflected by your Ent schema the ** desired** state, and the last state before your most recent changes the current state. There are two ways for Ent to determine the current state:

1. Replay the existing migration directory and inspect the schema (default)
2. Connect to an existing database and inspect the schema

We emphasize to use the first option, as it has the advantage for you to not have to connect to a production database to create a diff. In addition, this approach also works, if you have multiple deployments in different migration states. The first step is to enable the versioned migration feature by passing in the sql/versioned-migration feature flag. Depending on how you execute the Ent code generator, you have to use one of the two options:

With Ent CLI

If you are using the default go generate configuration, simply add the --feature sql/versioned-migration to the ent/generate.go file as follows:

package ent

//go:generate go run -mod=mod entgo.io/ent/cmd/ent generate --feature sql/versioned-migration ./schema

With entc package

If you are using the code generation package (e.g. if you are using an Ent extension), add the feature flag as follows:

//go:build ignore

package main

import (
	"log"

	"entgo.io/ent/entc"
	"entgo.io/ent/entc/gen"
)

func main() {
	err := entc.Generate("./schema", &gen.Config{
		Features: []gen.Feature{gen.FeatureVersionedMigration},
	})
	if err != nil {
		log.Fatalf("running ent codegen: %v", err)
	}
}

After regenerating the project, you have two ways to generate new migration files. First, there will be a new top-level method in the generated migrate package: NamedDiff. All you have to do is provide an Atlas dev database URL for the desired database dialect.

package main

import (
	"context"
	"log"

	"<project>/ent/migrate"

	atlas "ariga.io/atlas/sql/migrate"
	"entgo.io/ent/dialect"
	"entgo.io/ent/dialect/sql/schema"
	_ "github.com/go-sql-driver/mysql"
)

func main() {
	ctx := context.Background()
	// Create a local migration directory.
	dir, err := atlas.NewLocalDir("migrations")
	if err != nil {
		log.Fatalf("failed creating atlas migration directory: %v", err)
	}
	// Write migration diff.
	opts := []schema.MigrateOption{
		schema.WithDir(dir),                         // provide migration directory
		schema.WithMigrationMode(schema.ModeReplay), // provide migration mode
		schema.WithDialect(dialect.MySQL),           // Ent dialect to use
	}
	// Generate migrations using Atlas support for TiDB (note the Ent dialect option passed above).
	err = migrate.NamedDiff(ctx, "tidb://user:pass@localhost:3306/ent_dev", "my_migration", opts...)
	if err != nil {
		log.Fatalf("failed generating migration file: %v", err)
	}
}

You can then generate a new set of migration files by simply calling go run -mod=mod main.go.

:::info Note If you want to inspect an existing database and compute the diff against your Ent schema, pass in the ModeInspect migration mode. :::

A Word on Global Unique IDs

This section only applies to MySQL users using the global unique id feature.

When using the global unique ids, Ent allocates a range of 1<<32 integer values for each table. This is done by giving the first table an autoincrement starting value of 1, the second one the starting value 4294967296, the third one 8589934592, and so on. The order in which the tables receive the starting value is saved in an extra table called ent_types. With MySQL 5.6 and 5.7, the autoincrement starting value is only saved in memory (docs, InnoDB AUTO_INCREMENT Counter Initialization header) and re-calculated on startup by looking at the last inserted id for any table. Now, if you happen to have a table with no rows yet, the autoincrement starting value is set to 0 for every table without any entries. With the online migration feature this wasn't an issue, because the migration engine looked at the ent_types tables and made sure to update the counter, if it wasn't set correctly. However, with versioned migration, this is no longer the case. In oder to ensure, that everything is set up correctly after a server restart, make sure to call the VerifyTableRange method on the Atlas struct:

package main

import (
	"context"
	"log"

	"<project>/ent"
	"<project>/ent/migrate"
	"entgo.io/ent/dialect/sql"
	"entgo.io/ent/dialect/sql/schema"

	_ "github.com/go-sql-driver/mysql"
)

func main() {
	drv, err := sql.Open("mysql", "user:pass@tcp(localhost:3306)/ent")
	if err != nil {
		log.Fatalf("failed opening connection to mysql: %v", err)
	}
	defer drv.Close()
	// Verify the type allocation range.
	m, err := schema.NewMigrate(drv, nil)
	if err != nil {
		log.Fatalf("failed creating migrate: %v", err)
	}
	if err := m.VerifyTableRange(context.Background(), migrate.Tables); err != nil {
		log.Fatalf("failed verifyint range allocations: %v", err)
	}
	client := ent.NewClient(ent.Driver(drv))
	// ... do stuff with the client
}

:::caution Important After an upgrade to MySQL 8 from a previous version, you still have to run the method once to update the starting values. Since MySQL 8 the counter is no longer only stored in memory, meaning subsequent calls to the method are no longer needed after the first one. :::

Apply Migrations

The Atlas migration engine does not support applying the migration files onto a database yet, therefore to manage and execute the generated migration files, you have to rely on an external tool (or execute them by hand). By default, Atlas generates one "up" and one "down" migration file for the computed diff. These files are compatible with the popular golang-migrate/migrate package, and you can use that tool to manage the migrations in your deployments.

migrate -source file://migrations -database mysql://root:pass@tcp(localhost:3306)/test up

Moving from Auto-Migration to Versioned Migrations

In case you already have an Ent application in production and want to switch over from auto migration to the new versioned migration, you need to take some extra steps.

Create an initial migration file reflecting the currently deployed state

To do this make sure your schema definition is in sync with your deployed version(s). Then spin up an empty database and run the diff command once as described above. This will create the statements needed to create the current state of your schema graph. If you happened to have universal IDs enabled before, any deployment will have a special database table named ent_types. The above command will create the necessary SQL statements to create that table as well as its contents (similar to the following):

CREATE TABLE `users` (`id` integer NOT NULL PRIMARY KEY AUTOINCREMENT);
CREATE TABLE `groups` (`id` integer NOT NULL PRIMARY KEY AUTOINCREMENT);
INSERT INTO sqlite_sequence (name, seq) VALUES ("groups", 4294967296);
CREATE TABLE `ent_types` (`id` integer NOT NULL PRIMARY KEY AUTOINCREMENT, `type` text NOT NULL);
CREATE UNIQUE INDEX `ent_types_type_key` ON `ent_types` (`type`);
INSERT INTO `ent_types` (`type`) VALUES ('users'), ('groups');

In order to ensure to not break existing code, make sure the contents of that file are equal to the contents in the table present in the database you created the diff from. For example, if you consider the migration file from above (users,groups) but your deployed table looks like the one below (groups,users):

id	type
1	groups
2	users

You can see, that the order differs. In that case, you have to manually change both the entries in the generated migration file.

Configure the tool you use to manage migrations to consider this file as applied

In case of golang-migrate this can be done by forcing your database version as described here .

Use a Custom Formatter

Atlas' migration engine comes with great customizability. By the use of a custom Formatter you can generate the migration files in a format compatible with other migration management tools and Atlas has built-in support for the following four:

1. golang-migrate/migrate
2. pressly/goose
3. Flyway
4. Liquibase

:::note You need to have the latest master of Ent installed for this to be working.

go get -u entgo.io/ent@master

:::

package main

import (
	"context"
	"log"

	"<project>/ent/migrate"

	atlas "ariga.io/atlas/sql/migrate"
	"ariga.io/atlas/sql/sqltool"
	"entgo.io/ent/dialect"
	"entgo.io/ent/dialect/sql/schema"
	_ "github.com/go-sql-driver/mysql"
)

func main() {
	ctx := context.Background()
	// Create a local migration directory.
	dir, err := atlas.NewLocalDir("migrations")
	if err != nil {
		log.Fatalf("failed creating atlas migration directory: %v", err)
	}
	// Write migration diff.
	opts := []schema.MigrateOption{
		schema.WithDir(dir),                         // provide migration directory
		schema.WithMigrationMode(schema.ModeReplay), // provide migration mode
		schema.WithDialect(dialect.MySQL),           // Ent dialect to use
		// highlight-start
		// Choose one of the below.
		schema.WithFormatter(sqltool.GolangMigrateFormatter),
		schema.WithFormatter(sqltool.GooseFormatter),
		schema.WithFormatter(sqltool.FlywayFormatter),
		schema.WithFormatter(sqltool.LiquibaseFormatter),
		// highlight-end
	}
	// Generate migrations using Atlas support for TiDB (note the Ent dialect option passed above).
	err = migrate.NamedDiff(ctx, "tidb://user:pass@localhost:3306/ent_dev", "my_migration", opts...)
	if err != nil {
		log.Fatalf("failed generating migration file: %v", err)
	}
}

Note for using golang-migrate

If you use golang-migrate with MySQL, you need to add the multiStatements parameter to true as in the example below and then take the DSN we used in the documents with the param applied.

"user:password@tcp(host:port)/dbname?multiStatements=true"

Atlas migration directory integrity file

The Problem

Suppose you have multiple teams develop a feature in parallel and both of them need a migration. If Team A and Team B do not check in with each other, they might end up with a broken set of migration files (like adding the same table or column twice) since new files do not raise a merge conflict in a version control system like git. The following example demonstrates such behavior:

Assume both Team A and Team B add a new schema called User and generate a versioned migration file on their respective branch.

-- create "users" table
CREATE TABLE `users` (
    `id` bigint NOT NULL AUTO_INCREMENT,
    // highlight-start
    `team_a_col` INTEGER NOT NULL,
    // highlight-end
    PRIMARY KEY (`id`)
) CHARSET utf8mb4 COLLATE utf8mb4_bin;

-- create "users" table
CREATE TABLE `users` (
    `id` bigint NOT NULL AUTO_INCREMENT,
    // highlight-start
     `team_b_col` INTEGER NOT NULL,
    // highlight-end
     PRIMARY KEY (`id`)
) CHARSET utf8mb4 COLLATE utf8mb4_bin;

If they both merge their branch into master, git will not raise a conflict and everything seems fine. But attempting to apply the pending migrations will result in migration failure:

mysql> CREATE TABLE `users` (`id` bigint NOT NULL AUTO_INCREMENT, `team_a_col` INTEGER NOT NULL, PRIMARY KEY (`id`)) CHARSET utf8mb4 COLLATE utf8mb4_bin;
[2022-04-14 10:00:38] completed in 31 ms

mysql> CREATE TABLE `users` (`id` bigint NOT NULL AUTO_INCREMENT, `team_b_col` INTEGER NOT NULL, PRIMARY KEY (`id`)) CHARSET utf8mb4 COLLATE utf8mb4_bin;
[2022-04-14 10:00:48] [42S01][1050] Table 'users' already exists

Depending on the SQL this can potentially leave your database in a crippled state.

The Solution

Luckily, the Atlas migration engine offers a way to prevent concurrent creation of new migration files and guard against accidental changes in the migration history we call Migration Directory Integrity File, which simply is another file in your migration directory called atlas.sum. For the migration directory of team A it would look similar to this:

h1:KRFsSi68ZOarsQAJZ1mfSiMSkIOZlMq4RzyF//Pwf8A=
20220318104614_team_A.sql h1:EGknG5Y6GQYrc4W8e/r3S61Aqx2p+NmQyVz/2m8ZNwA=

The atlas.sum file contains the checksum of each migration file (implemented by a reverse, one branch merkle hash tree), and a sum of all files. Adding new files results in a change to the sum file, which will raise merge conflicts in most version controls systems. Let's see how we can use the Migration Directory Integrity File to detect the case from above automatically.

:::note Please note, that you need to have the Atlas CLI installed in your system for this to work, so make sure to follow the installation instructions before proceeding. :::

In previous versions of Ent, the integrity file was opt-in. But we think this is a very important feature that provides great value and safety to migrations. Therefore, generation of the sum file is now the default behavior and in the future we might even remove the option to disable this feature. For now, if you really want to remove integrity file generation, use the schema.DisableChecksum() option.

In addition to the usual .sql migration files the migration directory will contain the atlas.sum file. Every time you let Ent generate a new migration file, this file is updated for you. However, every manual change made to the mitration directory will render the migration directory and the atlas.sum file out-of-sync. With the Atlas CLI you can both check if the file and migration directory are in-sync, and fix it if not:

# If there is no output, the migration directory is in-sync.
atlas migrate validate --dir file://<path-to-your-migration-directory>

# If the migration directory and sum file are out-of-sync the Atlas CLI will tell you.
atlas migrate validate --dir file://<path-to-your-migration-directory>
Error: checksum mismatch

You have a checksum error in your migration directory.
This happens if you manually create or edit a migration file.
Please check your migration files and run

'atlas migrate hash --force'

to re-hash the contents and resolve the error.

exit status 1

If you are sure, that the contents in your migration files are correct, you can re-compute the hashes in the atlas.sum file:

# Recompute the sum file.
atlas migrate hash --dir file://<path-to-your-migration-directory> --force

Back to the problem above, if team A would land their changes on master first and team B would now attempt to land theirs, they'd get a merge conflict, as you can see in the example below:

You can add the atlas migrate validate call to your CI to have the migration directory checked continuously. Even if any team member would now forget to update the atlas.sum file after a manual edit, the CI would not go green, indicating a problem.