zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-04-30 12:07:51 +02:00
Code Issues Projects Releases Packages Wiki Activity

Update schema deployment doc and flyway tool (#363)

Browse source 
* Update schema deployment doc and flyway tool

Disabled Flyway Gradle tasks with side effects on Cloud SQL
instances. They can still be used on local databases.

Also switched Flyway Gradle tasks to get credentials from
new locations (in domain-registry-dev).

Updated README file on schema push process. Also reformatted
the entire file.
This commit is contained in:
Weimin Yu 2019-11-15 11:44:21 -05:00 • committed by GitHub
parent d2ebb591a2
commit bd27840068
2 changed files with 99 additions and 69 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

100
db/README.md
View file

@ -9,16 +9,16 @@ Nomulus uses the 'postgres' database in the 'public' schema. The following
users/roles are defined:
* postgres: the initial user is used for admin and schema deployment.
* In Cloud SQL, we do not control superusers. The initial 'postgres' user
is a regular user with create-role/create-db privileges. Therefore,
it is not possible to separate admin user and schema-deployment user.
* In Cloud SQL, we do not control superusers. The initial 'postgres' user
is a regular user with create-role/create-db privileges. Therefore, it
is not possible to separate admin user and schema-deployment user.
* readwrite is a role with read-write privileges on all data tables and
sequences. However, it does not have write access to admin tables. Nor
can it create new tables.
sequences. However, it does not have write access to admin tables. Nor can
it create new tables.
* The Registry server user is granted this role.
* readonly is a role with SELECT privileges on all tables.
* Reporting job user and individual human readers may be granted
this role.
* Reporting job user and individual human readers may be granted this
role.
### Schema DDL Scripts
@ -34,22 +34,28 @@ Below are the steps to submit a schema change:
2. Run the `nomulus generate_sql_schema` command to generate a new version of
`db-schema.sql.generated`. The full command line to do this is:
`./gradlew registryTool --args="-e localhost generate_sql_schema --start_postgresql -o /path/to/nomulus/db/src/main/resources/sql/schema/db-schema.sql.generated"`
3. Write an incremental DDL script that changes the existing schema to your
new one. The generated SQL file from the previous step should help. New
create table statements can be used as is, whereas alter table statements
should be written to change any existing tables.
`./gradlew registryTool --args="-e localhost generate_sql_schema
--start_postgresql -o
/path/to/nomulus/db/src/main/resources/sql/schema/db-schema.sql.generated"`
3. Write an incremental DDL script that changes the existing schema to your new
one. The generated SQL file from the previous step should help. New create
table statements can be used as is, whereas alter table statements should be
written to change any existing tables.
This script should be stored in a new file in the
`db/src/main/resources/sql/flyway` folder using the naming pattern
`V{id}__{description text}.sql`, where `{id}` is the next highest number
following the existing scripts in that folder. Note the double underscore in
the naming pattern.
4. Run the `:db:test` task from the Gradle root project. The SchemaTest will
fail because the new schema does not match the golden file.
5. Copy `db/build/resources/test/testcontainer/mount/dump.txt` to the golden file
`db/src/main/resources/sql/schema/nomulus.golden.sql`. Diff it against the
old version and verify that all changes are expected.
5. Copy `db/build/resources/test/testcontainer/mount/dump.txt` to the golden
file `db/src/main/resources/sql/schema/nomulus.golden.sql`. Diff it against
the old version and verify that all changes are expected.
6. Re-run the `:db:test` task. This time all tests should pass.
Relevant files (under db/src/main/resources/sql/schema/):
@ -66,23 +72,63 @@ example, when adding a new column to a table, we would deploy the change before
adding it to the relevant ORM class. Therefore, for a short time the golden file
will contain the new column while the generated one does not.
### Non-production Schema Push
### Schema Push
To manage schema in a non-production environment, use the 'flywayMigration'
task. You will need Cloud SDK and login once.
Currently Cloud SQL schema is released with the Nomulus server, and shares the
server release's tag (e.g., nomulus-20191101-RC00). Automatic schema push
process (to apply new changes in a released schema to the databases) has not
been set up yet, and new schema may be pushed manually on demand.
Presubmit and continuous-integration tests are being implemented to ensure
server/schema compatibility. Before the tests are activated, please look for
breaking changes before deploying a schema.
Released schema may be deployed using Cloud Build. Use the root project
directory as working directory, run the following shell snippets:
```shell
# Tags exist as folder names under gs://domain-registry-dev-deploy.
SCHEMA_TAG=
# Recognized environments are alpha, crash, sandbox and production
SQL_ENV=
# Deploy on cloud build. The --project is optional if domain-registry-dev
# is already your default project.
gcloud builds submit --config=release/cloudbuild-schema-deploy.yaml \
--substitutions=TAG_NAME=${SCHEMA_TAG},_ENV=${SQL_ENV} \
--project domain-registry-dev
# Verify by checking Flyway Schema History:
./gradlew :db:flywayInfo -PdbServer=${SQL_ENV}
```
#### Glass Breaking
If you need to deploy a schema off-cycle, try making a release first, then
deploy that release schema to Cloud SQL.
TODO(weiminyu): elaborate on different ways to push schema without a full
release.
#### Notes On Flyway
Please note: to run Flyway commands, you need Cloud SDK and need to log in once.
```shell
# One time login
gcloud auth login
# Deploy the current schema to alpha
gradlew :db:flywayMigrate -PdbServer=alpha
# Delete the entire schema in alpha
gradlew :db:flywayClean -PdbServer=alpha
```
The flywayMigrate task is idempotent. Repeated runs will not introduce problems.
The Flyway-based Cloud Build schema push process is safe in common scenarios:
* Repeatedly deploying the latest schema is safe. All duplicate runs become
NOP.
* Accidentally deploying a past schema is safe. Flyway will not undo
incremental changes not reflected in the deployed schema.
* Concurrent deployment runs are safe. Flyway locks its own metadata table,
serializing deployment runs without affecting normal accesses.
#### Schema Push to Local Database
The Flyway tasks may also be used to deploy to local instances, e.g, your own
test instance. E.g.,
@ -95,7 +141,3 @@ gradlew :db:flywayMigrate -PdbServer=192.168.9.2 -PdbPassword=domain-registry
gradlew :db:flywayMigrate -PdbServer=192.168.9.2:5432 -PdbUser=postgres \
-PdbPassword=domain-registry
```
### Production Schema Deployment
Schema deployment to production and sandbox is under development.

64
db/build.gradle
View file

@ -31,24 +31,8 @@ ext {
def dbServer = findProperty(dbServerProperty).toString().toLowerCase()
def dbName = findProperty(dbNameProperty)
reconfirmRestrictedDbEnv = {
if (!restrictedDbEnv.contains(dbServer)) {
return
}
// For restricted environments, ask the user to type again to confirm.
// The following statement uses Gradle internal API to get around the
// missing console bug when Gradle Daemon is in use. Another option is
// to use the ant.input task. For details please refer to
// https://github.com/gradle/gradle/issues/1251.
def dbServerAgain = services.get(UserInputHandler.class).askQuestion(
"""\
Are you sure? Operating on ${dbServer} from desktop is unsafe.
Please type '${dbServer}' again to proceed: """.stripIndent(),
'').trim()
if (dbServer != dbServerAgain) {
throw new RuntimeException(
"Failed to confirm for restricted database environment. Operation aborted.")
}
isCloudSql = {
return allDbEnv.contains(dbServer)
}
getAccessInfoByHostPort = { hostAndPort ->
@ -87,18 +71,14 @@ ext {
// production. The role parameter may be superuser. (More roles will be added
// later).
getCloudSqlCredential = { env, role ->
env = env == 'production' ? '' : "-${env}"
def keyProject = env == '-crash'
? 'domain-registry-crash-kms-keys'
: "domain-registry${env}-keys"
def command =
"""gsutil cp \
gs://domain-registry${env}-cloudsql-credentials/${role}_credential.enc - | \
gs://domain-registry-dev-deploy/cloudsql-credentials/${env}/${role}_credential.enc - | \
base64 -d | \
gcloud kms decrypt --location global --keyring nomulus \
--key sql-credentials-on-gcs-key --plaintext-file=- \
gcloud kms decrypt --location global --keyring nomulus-tool-keyring \
--key nomulus-tool-key --plaintext-file=- \
--ciphertext-file=- \
--project=${keyProject}"""
--project=domain-registry-dev"""
return execInBash(command, '/tmp')
}
@ -132,7 +112,6 @@ publishing {
}
}
flyway {
def accessInfo = project.ext.getJdbcAccessInfo()
@ -144,13 +123,6 @@ flyway {
locations = [ "classpath:sql/flyway" ]
}
tasks.flywayMigrate.dependsOn(
tasks.create('confirmMigrateOnRestrictedDb') {
doLast {
project.ext.reconfirmRestrictedDbEnv()
}
})
dependencies {
def deps = rootProject.dependencyMap
@ -170,7 +142,23 @@ dependencies {
testCompile project(':third_party')
}
// Ensure that resources are rebuilt before running Flyway tasks
tasks
.findAll { task -> task.group.equals('Flyway')}
.collect { task -> task.dependsOn('buildNeeded') }
flywayValidate.dependsOn('buildNeeded')
if (ext.isCloudSql()) {
// Disable dangerous Flyway tasks. Only allow info and validate.
tasks.findAll { task -> task.group.equals('Flyway')}.each {
if (it.name == 'flywayMigrate') {
it.doFirst {
throw new UnsupportedOperationException(
""" \
FlywayMigrate is disabled. See README.md for schema deployment
instructions.""".stripIndent())
}
} else if (it.name != 'flywayInfo' && it.name != 'flywayValidate') {
it.doFirst {
throw new UnsupportedOperationException(
"${it.name} from commandline is not allowed.")
}
}
}
}