From 8916d6fbaf1e7b75577ede8105d26b5603d114be Mon Sep 17 00:00:00 2001 From: Ben McIlwain Date: Fri, 13 Aug 2021 17:21:38 -0400 Subject: [PATCH] Add instructions for two-step DB schema updates (#1283) * Add instructions for two-step DB schema updates These expanded steps are required by the recent enabling of the SQL integration test suite. --- db/README.md | 39 +++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/db/README.md b/db/README.md index 1fc9d5d70..d71aa4e81 100644 --- a/db/README.md +++ b/db/README.md @@ -3,7 +3,7 @@ This project contains Nomulus's Cloud SQL schema and schema-deployment utilities. -### ER Diagrams +### Entity Relationship (ER) diagrams The following links are the ER diagrams generated from the current SQL schema: @@ -14,7 +14,7 @@ shows all columns, foreign keys and indexes. shows only significant columns, such as primary and foreign key columns, and columns that are part of unique indexes. -### Database Roles and Privileges +### Database roles and privileges Nomulus uses the 'postgres' database in the 'public' schema. The following users/roles are defined: @@ -31,14 +31,18 @@ users/roles are defined: * Reporting job user and individual human readers may be granted this role. -### Schema DDL Scripts +### How to update the schema Currently we use Flyway for schema deployment. Versioned incremental update scripts are organized in the src/main/resources/sql/flyway folder. A Flyway 'migration' task examines the target database instance, and makes sure that only changes not yet deployed are pushed. -Below are the steps to submit a schema change: +Because we have SQL integration tests enabled to ensure that deployments are +rollback-safe, which prevent Java code from executing against a version of the +schema it is incompatible with, you will need to commit your schema additions in +two separate PRs, with a wait for a deployment in-between, as explained in the +following steps: 1. Make your changes to entity classes, remembering to add new ones to `core/src/main/resources/META-INF/persistence.xml` so they'll be picked up. @@ -76,6 +80,16 @@ Below are the steps to submit a schema change: You'll want to have a look at the diffs in the golden schema to verify that all changes are intentional. +6. Now, split your outstanding changes into two PRs. The first PR should _only_ + include your new Flyway version `.sql` file, its addition to the `flyway.txt` + index, changes to the `nomulus.golden.sql` schema file, and changes to the + Entity Relationship diagram `.html` files. The second PR should include + everything else, including _all_ changes to `.java` files and the + `db-schema.sql.generated` changes that derive from them. +7. Submit the first PR and wait until it is successfully deployed to production, + then submit the second PR. Note, if you are removing things from the schema + (rather than adding them), then these PRs should be in the opposite order: + Java changes first, then SQL changes afterwards. Relevant files (under `db/src/main/resources/sql/schema/`): @@ -91,7 +105,16 @@ example, when adding a new column to a table, we would deploy the change before adding it to the relevant ORM class. Therefore, for a short time the golden file will contain the new column while the generated one does not. -### Schema Push +Note that, when making schema changes, you _cannot_ add a new `NOT NULL` column +to an existing table that does not have a default value, or make any other +similar addition of a constraint that will be violated by existing data. If you +wish to rename a column, you must first add a new column with the desired name, +copy over its contents using a `@PostLoad` action in Java, re-save all rows, +update the Java to no longer contain the old column, wait for a deployment, and +then remove the old column. A rename operation requires the most complicated +series of steps to complete, as it is effectively an add followed by a remove. + +### Schema push Currently Cloud SQL schema is released with the Nomulus server, and shares the server release's tag (e.g., `nomulus-20191101-RC00`). Automatic schema push @@ -126,7 +149,7 @@ following command to deploy the local schema, ./nom_build :db:flywayMigrate --dbServer=[alpha|crash] --environment=[alpha|crash] ``` -#### Glass Breaking +#### Glass breaking If you need to deploy a schema off-cycle, try making a release first, then deploy that release schema to Cloud SQL. @@ -134,7 +157,7 @@ deploy that release schema to Cloud SQL. TODO(weiminyu): elaborate on different ways to push schema without a full release. -#### Notes On Flyway +#### Notes on Flyway Please note: to run Flyway commands, you need Cloud SDK and need to log in once. @@ -154,7 +177,7 @@ The Flyway-based Cloud Build schema push process is safe in common scenarios: * Concurrent deployment runs are safe. Flyway locks its own metadata table, serializing deployment runs without affecting normal accesses. -#### Schema Push to Local Database +#### Schema push to local database The Flyway tasks may also be used to deploy to local instances, e.g, your own test instance. E.g.,