diff --git a/doc/md/versioned-migrations.md b/doc/md/versioned-migrations.md index d6b0d39c5..b8a47e486 100644 --- a/doc/md/versioned-migrations.md +++ b/doc/md/versioned-migrations.md @@ -10,10 +10,13 @@ tool you like (like golang-migrate, Flyway, liquibase). ![atlas-versioned-migration-process](https://entgo.io/images/assets/migrate-atlas-versioned.png) -## Configuration +## Generating Versioned Migration Files -In order to have Ent make the necessary changes to your code, you have to enable this feature with one of the two -options: +### From Client + +If you want to use an instantiated Ent client to create new migration files, you have to enable the versioned +migrations feature flag in order to have Ent make the necessary changes to the generated code. Depending on how you +execute the Ent code generator, you have to use one of the two options: 1. If you are using the default go generate configuration, simply add the `--feature sql/versioned-migration` to the `ent/generate.go` file as follows: @@ -47,12 +50,8 @@ func main() { } ``` -## Generating Versioned Migration Files - -### From Client - After regenerating the project, there will be an extra `Diff` method on the Ent client that you can use to inspect the -connected database, compare it with the schema definitions and create sql statements needed to migrate the database to +connected database, compare it with the schema definitions and create SQL statements needed to migrate the database to the graph. ```go @@ -95,7 +94,7 @@ You can then create a new set of migration files by simply calling `go run -mod= ### From Graph You can also generate new migration files without an instantiated Ent client. This can be useful if you want to make the -migration file creation part of a go generate workflow. +migration file creation part of a go generate workflow. Note, that in this case enabling the feature flag is optional. ```go package main diff --git a/doc/website/blog/2022-03-14-announcing-versioned-migrations.md b/doc/website/blog/2022-03-14-announcing-versioned-migrations.md new file mode 100644 index 000000000..98b5bac56 --- /dev/null +++ b/doc/website/blog/2022-03-14-announcing-versioned-migrations.md @@ -0,0 +1,364 @@ +--- +title: Announcing Versioned Migrations Authoring +author: MasseElch +authorURL: "https://github.com/masseelch" +authorImageURL: "https://avatars.githubusercontent.com/u/12862103?v=4" +image: "https://entgo.io/images/assets/migrate/versioned-share.png" +--- + +When [Ariel](https://github.com/a8m) released Ent v0.10.0 at the end of January, +he [introduced](2022-01-20-announcing-new-migration-engine.md) a new migration engine for Ent based on another +open-source project called [Atlas](https://github.com/ariga/atlas). + +Initially, Atlas supported a style of managing database schemas that we call "declarative migrations". With declarative +migrations, the desired state of the database schema is given as input to the migration engine, which plans and executes +a set of actions to change the database to its desired state. This approach has been popularized in the field of +cloud native infrastructure by projects such as Kubernetes and Terraform. It works great in many cases, in +fact it has served the Ent framework very well in the past few years. However, database migrations are a very sensitive +topic, and many projects require a more controlled approach. + +For this reason, most industry standard solutions, like [Flyway](https://flywaydb.org/) +, [Liquibase](https://liquibase.org/), or [golang-migrate/migrate](https://github.com/golang-migrate/migrate) (which is +common in the Go ecosystem), support a workflow that they call "versioned migrations". + +With versioned migrations (sometimes called "change base migrations") instead of describing the desired state ("what the +database should look like"), you describe the changes itself ("how to reach the state"). Most of the time this is done +by creating a set of SQL files containing the statements needed. Each of the files is assigned a unique version and a +description of the changes. Tools like the ones mentioned earlier are then able to interpret the migration files and to +apply (some of) them in the correct order to transition to the desired database structure. + +In this post, I want to showcase a new kind of migration workflow that has recently been added to Atlas and Ent. We call +it "versioned migration authoring" and it's an attempt to combine the simplicity and expressiveness of the declarative +approach with the safety and explicitness of versioned migrations. With versioned migration authoring, users still +declare their desired state and use the Atlas engine to plan a safe migration from the existing to the new state. +However, instead of coupling the planning and execution, it is instead written into a file which can be checked into +source control, fine-tuned manually and reviewed in normal code review processes. + +As an example, I will demonstrate the workflow with `golang-migrate/migrate`. + +### Getting Started + +The very first thing to do, is to make sure you have an up-to-date Ent version: + +```shell +go get -u entgo.io/ent@master +``` + +There are two ways to have Ent generate migration files for schema changes. The first one is to use an instantiated Ent +client and the second one to generate the changes from a parsed schema graph. This post will take the second approach, +if you want to learn how to use the first one you can have a look at +the [documentation](./docs/versioned-migrations#from-client). + +### Generating Versioned Migration Files + +Since we have enabled the versioned migrations feature now, let's create a small schema and generate the initial set of +migration files. Consider the following schema for a fresh Ent project: + +```go title="ent/schema/user.go" +package schema + +import ( + "entgo.io/ent" + "entgo.io/ent/schema/field" + "entgo.io/ent/schema/index" +) + +// User holds the schema definition for the User entity. +type User struct { + ent.Schema +} + +// Fields of the User. +func (User) Fields() []ent.Field { + return []ent.Field{ + field.String("username"), + } +} + +// Indexes of the User. +func (User) Indexes() []ent.Index { + return []ent.Index{ + index.Fields("username").Unique(), + } +} + +``` + +As I stated before, we want to use the parsed schema graph to compute the difference between our schema and the +connected database. Here is an example of a (semi-)persistent MySQL docker container to use if you want to follow along: + +```shell +docker run --rm --name ent-versioned-migrations --detach --env MYSQL_ROOT_PASSWORD=pass --env MYSQL_DATABASE=ent -p 3306:3306 mysql +``` + +Once you are done, you can shut down the container and remove all resources with `docker stop ent-versioned-migrations`. + +Now, let's create a small function that loads the schema graph and generates the migration files. Create a new Go file +named `main.go` and copy the following contents: + +```go title="main.go" +package main + +import ( + "context" + "log" + "os" + + "ariga.io/atlas/sql/migrate" + "entgo.io/ent/dialect/sql" + "entgo.io/ent/dialect/sql/schema" + "entgo.io/ent/entc" + "entgo.io/ent/entc/gen" + _ "github.com/go-sql-driver/mysql" +) + +func main() { + // We need a name for the new migration file. + if len(os.Args) < 2 { + log.Fatalln("no name given") + } + // Create a local migration directory. + dir, err := migrate.NewLocalDir("migrations") + if err != nil { + log.Fatalln(err) + } + // Load the graph. + graph, err := entc.LoadGraph("./ent/schema", &gen.Config{}) + if err != nil { + log.Fatalln(err) + } + tbls, err := graph.Tables() + if err != nil { + log.Fatalln(err) + } + // Open connection to the database. + drv, err := sql.Open("mysql", "root:pass@tcp(localhost:3306)/ent") + if err != nil { + log.Fatalln(err) + } + // Inspect the current database state and compare it with the graph. + m, err := schema.NewMigrate(drv, schema.WithDir(dir)) + if err != nil { + log.Fatalln(err) + } + if err := m.NamedDiff(context.Background(), os.Args[1], tbls...); err != nil { + log.Fatalln(err) + } +} +``` + +All we have to do now is create the migration directory and execute the above Go file: + +```shell +mkdir migrations +go run -mod=mod main.go initial +``` + +You will now see two new files in the `migrations` directory: `_initial.down.sql` +and `_initial.up.sql`. The `x.up.sql` files are used to create the database version `x` and `x.down.sql` to +roll back to the previous version. + +```sql title="_initial.up.sql" +CREATE TABLE `users` (`id` bigint NOT NULL AUTO_INCREMENT, `username` varchar(191) NOT NULL, PRIMARY KEY (`id`), UNIQUE INDEX `user_username` (`username`)) CHARSET utf8mb4 COLLATE utf8mb4_bin; +``` + +```sql title="_initial.down.sql" +DROP TABLE `users`; +``` + +### Applying Migrations + +To apply these migrations on your database, install the `golang-migrate/migrate` tool as described in +their [README](https://github.com/golang-migrate/migrate/blob/master/cmd/migrate/README.md). Then run the following +command to check if everything went as it should. + +```shell +migrate -help +``` +```text +Usage: migrate OPTIONS COMMAND [arg...] + migrate [ -version | -help ] + +Options: + -source Location of the migrations (driver://url) + -path Shorthand for -source=file://path + -database Run migrations against this database (driver://url) + -prefetch N Number of migrations to load in advance before executing (default 10) + -lock-timeout N Allow N seconds to acquire database lock (default 15) + -verbose Print verbose logging + -version Print version + -help Print usage + +Commands: + create [-ext E] [-dir D] [-seq] [-digits N] [-format] NAME + Create a set of timestamped up/down migrations titled NAME, in directory D with extension E. + Use -seq option to generate sequential up/down migrations with N digits. + Use -format option to specify a Go time format string. + goto V Migrate to version V + up [N] Apply all or N up migrations + down [N] Apply all or N down migrations + drop Drop everything inside database + force V Set version V but don't run migration (ignores dirty state) + version Print current migration version +``` + +Now we can execute our initial migration and sync the database with our schema: + +```shell +migrate -source 'file://migrations' -database 'mysql://root:pass@tcp(localhost:3306)/ent' up +``` +```text +/u initial (349.256951ms) +``` + +### Workflow + +To demonstrate the usual workflow when using versioned migrations we will both edit our schema graph and generate the +migration changes for it, and manually create a set of migration files to seed the database with some data. First, we +will add a Group schema and a many-to-many relation to the existing User schema, next create an admin Group with an +admin User in it. Go ahead and make the following changes: + +```go title="ent/schema/user.go" {22-28} +package schema + +import ( + "entgo.io/ent" + "entgo.io/ent/schema/edge" + "entgo.io/ent/schema/field" + "entgo.io/ent/schema/index" +) + +// User holds the schema definition for the User entity. +type User struct { + ent.Schema +} + +// Fields of the User. +func (User) Fields() []ent.Field { + return []ent.Field{ + field.String("username"), + } +} + +// Edges of the User. +func (User) Edges() []ent.Edge { + return []ent.Edge{ + edge.From("groups", Group.Type). + Ref("users"), + } +} + +// Indexes of the User. +func (User) Indexes() []ent.Index { + return []ent.Index{ + index.Fields("username").Unique(), + } +} +``` + +```go title="ent/schema/group.go" +package schema + +import ( + "entgo.io/ent" + "entgo.io/ent/schema/edge" + "entgo.io/ent/schema/field" + "entgo.io/ent/schema/index" +) + +// Group holds the schema definition for the Group entity. +type Group struct { + ent.Schema +} + +// Fields of the Group. +func (Group) Fields() []ent.Field { + return []ent.Field{ + field.String("name"), + } +} + +// Edges of the Group. +func (Group) Edges() []ent.Edge { + return []ent.Edge{ + edge.To("users", User.Type), + } +} + +// Indexes of the Group. +func (Group) Indexes() []ent.Index { + return []ent.Index{ + index.Fields("name").Unique(), + } +} +``` +Once the schema is updated, create a new set of migration files. + +```shell +go run -mod=mod main.go add_group_schema +``` + +Once again there will be two new files in the `migrations` directory: `_add_group_schema.down.sql` +and `_add_group_schema.up.sql`. + +```sql title="_add_group_schema.up.sql" +CREATE TABLE `groups` (`id` bigint NOT NULL AUTO_INCREMENT, `name` varchar(191) NOT NULL, PRIMARY KEY (`id`), UNIQUE INDEX `group_name` (`name`)) CHARSET utf8mb4 COLLATE utf8mb4_bin; +CREATE TABLE `group_users` (`group_id` bigint NOT NULL, `user_id` bigint NOT NULL, PRIMARY KEY (`group_id`, `user_id`), CONSTRAINT `group_users_group_id` FOREIGN KEY (`group_id`) REFERENCES `groups` (`id`) ON DELETE CASCADE, CONSTRAINT `group_users_user_id` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE) CHARSET utf8mb4 COLLATE utf8mb4_bin; +``` + +```sql title="_add_group_schema.down.sql" +DROP TABLE `group_users`; +DROP TABLE `groups`; +``` + +Now you can either edit the generated files to add the seed data or create new files for it. I chose the latter: + +```shell +migrate create -format unix -ext sql -dir migrations seed_admin +``` +```text +[...]/ent-versioned-migrations/migrations/_seed_admin.up.sql +[...]/ent-versioned-migrations/migrations/_seed_admin.down.sql +``` + +You can now edit those files and add statements to create an admin Group and User. + +```sql title="migrations/_seed_admin.up.sql" +INSERT INTO `groups` (`id`, `name`) VALUES (1, 'Admins'); +INSERT INTO `users` (`id`, `username`) VALUES (1, 'admin'); +INSERT INTO `group_users` (`group_id`, `user_id`) VALUES (1, 1); +``` + +```sql title="migrations/_seed_admin.down.sql" +DELETE FROM `group_users` where `group_id` = 1 and `user_id` = 1; +DELETE FROM `groups` where id = 1; +DELETE FROM `users` where id = 1; +``` + +Apply the migrations once more, and you are done: + +```shell +migrate -source file://migrations -database 'mysql://root:pass@tcp(localhost:3306)/ent' up +``` + +```text +/u add_group_schema (417.434415ms) +/u seed_admin (674.189872ms) +``` + +### Wrapping Up + +In this post, we demonstrated the general workflow when using Ent Versioned Migrations with `golang-migate/migrate`. We +created a small example schema, generated the migration files for it and learned how to apply them. We now know the +workflow and how to add custom migration files. + +Have questions? Need help with getting started? Feel free to [join our Slack channel](https://entgo.io/docs/slack/). + +:::note For more Ent news and updates: + +- Subscribe to our [Newsletter](https://www.getrevue.co/profile/ent) +- Follow us on [Twitter](https://twitter.com/entgo_io) +- Join us on #ent on the [Gophers Slack](https://entgo.io/docs/slack) +- Join us on the [Ent Discord Server](https://discord.gg/qZmPgTE6RX) + +:::