Update schema deployment doc and flyway tool (#363)

* Update schema deployment doc and flyway tool

Disabled Flyway Gradle tasks with side effects on Cloud SQL
instances. They can still be used on local databases.

Also switched Flyway Gradle tasks to get credentials from
new locations (in domain-registry-dev).

Updated README file on schema push process. Also reformatted
the entire file.

db/README.md

@@ -10,15 +10,15 @@ users/roles are defined:
 
 *   postgres: the initial user is used for admin and schema deployment.
     *   In Cloud SQL, we do not control superusers. The initial 'postgres' user
-       is a regular user with create-role/create-db privileges. Therefore,
+        is a regular user with create-role/create-db privileges. Therefore, it
-       it is not possible to separate admin user and schema-deployment user.
+        is not possible to separate admin user and schema-deployment user.
 *   readwrite is a role with read-write privileges on all data tables and
-    sequences. However, it does not have write access to admin tables. Nor
+    sequences. However, it does not have write access to admin tables. Nor can
-    can it create new tables.
+    it create new tables.
     *   The Registry server user is granted this role.
 *   readonly is a role with SELECT privileges on all tables.
-    *   Reporting job user and individual human readers may be granted
+    *   Reporting job user and individual human readers may be granted this
-        this role.
+        role.
 
 ### Schema DDL Scripts
 
@@ -34,22 +34,28 @@ Below are the steps to submit a schema change:
 2.  Run the `nomulus generate_sql_schema` command to generate a new version of
     `db-schema.sql.generated`. The full command line to do this is:
 
-    `./gradlew registryTool --args="-e localhost generate_sql_schema --start_postgresql -o /path/to/nomulus/db/src/main/resources/sql/schema/db-schema.sql.generated"`
+    `./gradlew registryTool --args="-e localhost generate_sql_schema
-3.  Write an incremental DDL script that changes the existing schema to your
+    --start_postgresql -o
-    new one. The generated SQL file from the previous step should help. New 
+    /path/to/nomulus/db/src/main/resources/sql/schema/db-schema.sql.generated"`
-    create table statements can be used as is, whereas alter table statements
+
-    should be written to change any existing tables.
+3.  Write an incremental DDL script that changes the existing schema to your new
+    one. The generated SQL file from the previous step should help. New create
+    table statements can be used as is, whereas alter table statements should be
+    written to change any existing tables.
 
     This script should be stored in a new file in the
     `db/src/main/resources/sql/flyway` folder using the naming pattern
     `V{id}__{description text}.sql`, where `{id}` is the next highest number
     following the existing scripts in that folder. Note the double underscore in
     the naming pattern.
+
 4.  Run the `:db:test` task from the Gradle root project. The SchemaTest will
     fail because the new schema does not match the golden file.
-5.  Copy `db/build/resources/test/testcontainer/mount/dump.txt` to the golden file
+
-    `db/src/main/resources/sql/schema/nomulus.golden.sql`. Diff it against the
+5.  Copy `db/build/resources/test/testcontainer/mount/dump.txt` to the golden
-    old version and verify that all changes are expected.
+    file `db/src/main/resources/sql/schema/nomulus.golden.sql`. Diff it against
+    the old version and verify that all changes are expected.
+
 6.  Re-run the `:db:test` task. This time all tests should pass.
 
 Relevant files (under db/src/main/resources/sql/schema/):
@@ -66,23 +72,63 @@ example, when adding a new column to a table, we would deploy the change before
 adding it to the relevant ORM class. Therefore, for a short time the golden file
 will contain the new column while the generated one does not.
 
-### Non-production Schema Push
+### Schema Push
 
-To manage schema in a non-production environment, use the 'flywayMigration'
+Currently Cloud SQL schema is released with the Nomulus server, and shares the
-task. You will need Cloud SDK and login once.
+server release's tag (e.g., nomulus-20191101-RC00). Automatic schema push
+process (to apply new changes in a released schema to the databases) has not
+been set up yet, and new schema may be pushed manually on demand.
+
+Presubmit and continuous-integration tests are being implemented to ensure
+server/schema compatibility. Before the tests are activated, please look for
+breaking changes before deploying a schema.
+
+Released schema may be deployed using Cloud Build. Use the root project
+directory as working directory, run the following shell snippets:
+
+```shell
+# Tags exist as folder names under gs://domain-registry-dev-deploy.
+SCHEMA_TAG=
+# Recognized environments are alpha, crash, sandbox and production
+SQL_ENV=
+# Deploy on cloud build. The --project is optional if domain-registry-dev
+# is already your default project.
+gcloud builds submit --config=release/cloudbuild-schema-deploy.yaml \
+    --substitutions=TAG_NAME=${SCHEMA_TAG},_ENV=${SQL_ENV} \
+    --project domain-registry-dev
+# Verify by checking Flyway Schema History:
+./gradlew :db:flywayInfo -PdbServer=${SQL_ENV}
+```
+
+#### Glass Breaking
+
+If you need to deploy a schema off-cycle, try making a release first, then
+deploy that release schema to Cloud SQL.
+
+TODO(weiminyu): elaborate on different ways to push schema without a full
+release.
+
+#### Notes On Flyway
+
+Please note: to run Flyway commands, you need Cloud SDK and need to log in once.
 
 ```shell
 # One time login
 gcloud auth login
-
-# Deploy the current schema to alpha
-gradlew :db:flywayMigrate -PdbServer=alpha
-
-# Delete the entire schema in alpha
-gradlew :db:flywayClean -PdbServer=alpha
 ```
 
-The flywayMigrate task is idempotent. Repeated runs will not introduce problems.
+The Flyway-based Cloud Build schema push process is safe in common scenarios:
+
+*   Repeatedly deploying the latest schema is safe. All duplicate runs become
+    NOP.
+
+*   Accidentally deploying a past schema is safe. Flyway will not undo
+    incremental changes not reflected in the deployed schema.
+
+*   Concurrent deployment runs are safe. Flyway locks its own metadata table,
+    serializing deployment runs without affecting normal accesses.
+
+#### Schema Push to Local Database
 
 The Flyway tasks may also be used to deploy to local instances, e.g, your own
 test instance. E.g.,
@@ -95,7 +141,3 @@ gradlew :db:flywayMigrate -PdbServer=192.168.9.2 -PdbPassword=domain-registry
 gradlew :db:flywayMigrate -PdbServer=192.168.9.2:5432 -PdbUser=postgres \
     -PdbPassword=domain-registry
 ```
-
-### Production Schema Deployment
-
-Schema deployment to production and sandbox is under development.

db/build.gradle

@@ -31,24 +31,8 @@ ext {
   def dbServer = findProperty(dbServerProperty).toString().toLowerCase()
   def dbName = findProperty(dbNameProperty)
 
-  reconfirmRestrictedDbEnv = {
+  isCloudSql = {
-    if (!restrictedDbEnv.contains(dbServer)) {
+    return allDbEnv.contains(dbServer)
-      return
-    }
-    // For restricted environments, ask the user to type again to confirm.
-    // The following statement uses Gradle internal API to get around the
-    // missing console bug when Gradle Daemon is in use. Another option is
-    // to use the ant.input task. For details please refer to
-    // https://github.com/gradle/gradle/issues/1251.
-    def dbServerAgain = services.get(UserInputHandler.class).askQuestion(
-        """\
-           Are you sure? Operating on ${dbServer} from desktop is unsafe.
-           Please type '${dbServer}' again to proceed: """.stripIndent(),
-        '').trim()
-    if (dbServer != dbServerAgain) {
-      throw new RuntimeException(
-          "Failed to confirm for restricted database environment. Operation aborted.")
-    }
   }
 
   getAccessInfoByHostPort = { hostAndPort ->
@@ -87,18 +71,14 @@ ext {
   // production. The role parameter may be superuser. (More roles will be added
   // later).
   getCloudSqlCredential = { env, role ->
-    env = env == 'production' ? '' : "-${env}"
-    def keyProject = env == '-crash'
-        ? 'domain-registry-crash-kms-keys'
-        : "domain-registry${env}-keys"
     def command =
         """gsutil cp \
-           gs://domain-registry${env}-cloudsql-credentials/${role}_credential.enc - | \
+           gs://domain-registry-dev-deploy/cloudsql-credentials/${env}/${role}_credential.enc - | \
            base64 -d | \
-           gcloud kms decrypt --location global --keyring nomulus \
+           gcloud kms decrypt --location global --keyring nomulus-tool-keyring \
-           --key sql-credentials-on-gcs-key --plaintext-file=- \
+           --key nomulus-tool-key --plaintext-file=- \
            --ciphertext-file=- \
-           --project=${keyProject}"""
+           --project=domain-registry-dev"""
 
     return execInBash(command, '/tmp')
   }
@@ -132,7 +112,6 @@ publishing {
   }
 }
 
-
 flyway {
   def accessInfo = project.ext.getJdbcAccessInfo()
 
@@ -144,13 +123,6 @@ flyway {
   locations = [ "classpath:sql/flyway" ]
 }
 
-tasks.flywayMigrate.dependsOn(
-    tasks.create('confirmMigrateOnRestrictedDb') {
-      doLast {
-        project.ext.reconfirmRestrictedDbEnv()
-      }
-    })
-
 dependencies {
   def deps = rootProject.dependencyMap
 
@@ -170,7 +142,23 @@ dependencies {
   testCompile project(':third_party')
 }
 
-// Ensure that resources are rebuilt before running Flyway tasks
+flywayValidate.dependsOn('buildNeeded')
-tasks
+
-    .findAll { task -> task.group.equals('Flyway')}
+if (ext.isCloudSql()) {
-    .collect { task -> task.dependsOn('buildNeeded') }
+  // Disable dangerous Flyway tasks. Only allow info and validate.
+  tasks.findAll { task -> task.group.equals('Flyway')}.each {
+    if (it.name == 'flywayMigrate') {
+      it.doFirst {
+        throw new UnsupportedOperationException(
+            """ \
+              FlywayMigrate is disabled. See README.md for schema deployment
+              instructions.""".stripIndent())
+      }
+    } else if (it.name != 'flywayInfo' && it.name != 'flywayValidate') {
+      it.doFirst {
+        throw new UnsupportedOperationException(
+            "${it.name} from commandline is not allowed.")
+      }
+    }
+  }
+}