Merge pull request #161 from cisagov/nmb/migration-strategy

Document database migration strategy

docs/architecture/decisions/0013-database-migration.md

@@ -0,0 +1,38 @@
+# 13. Use CloudFoundry tasks and Github Actions to manually run migrations
+
+Date: 2022-10-04
+
+## Status
+
+Accepted
+
+## Context
+
+A database-backed web application needs a way to change the “schema” or
+structure of the database (adding/dropping tables, adding/dropping/altering
+columns of tables). These changes are called “migrations”. They are checked-in
+to the Github repository along with the code that uses the changed database.
+When that new code is deployed, it will usually run, but may give errors until
+the matching migrations have been applied to the database.
+
+## Decision
+
+We will not run the checked-in migrations automatically as part of deploys. We
+will manually apply database migrations to our Cloud.gov environments using
+Cloudfoundry’s `run-task` command to run Django’s `manage.py migrate` command.
+For the `unstable` environment, we can run the `cf run-task . . .` command
+directly. For the `staging` environment we have a Github Actions workflow
+called “Run Migrations”
+<https://github.com/cisagov/getgov/actions/workflows/migrate.yaml> that will
+run that `cf` command using the credentials for `staging` that are only in the
+Github Actions configuration. 
+
+## Consequences
+
+Not automatically applying database migrations before running new code means
+that the site might stop working briefly until the database migrations are
+applied. This trade-off feels worth the gain in control that we get by not
+having migrations run unexpectedly.
+
+Some of these consequences could be mitigated by developer attention to the
+impacts in the unstable and/or staging environment.

docs/developer/database-access.md

@@ -0,0 +1,77 @@
+# Working with Cloud.gov Databases
+
+You can connect to a Cloud.gov database using the
+[cf-service-connect](https://github.com/cloud-gov/cf-service-connect) plugin.
+After installing it, use the command
+
+```shell
+cf connect-to-service getgov-unstable getgov-unstable-databse
+```
+
+to get a `psql` shell on the `unstable` environment's database.
+
+## Running Migrations
+
+When new code changes the database schema, we need to apply Django's migrations.
+We can run these using CloudFoundry's tasks to run the `manage.py migrate`
+command in the correct environment. For the `unstable` environment, developers
+can manually run the task with
+
+```shell
+cf run-task getgov-unstable --command 'python manage.py migrate' --name migrate
+```
+
+For the `staging` environment, developers don't have credentials so we need to
+run that command using Github Actions. Go to
+<https://github.com/cisagov/getgov/actions/workflows/migrate.yaml> and select
+the "Run workflow" button, making sure that `staging` is selected.
+
+## Dropping and re-creating the database
+
+For `unstable`, it might be necessary to start the database over from scratch.
+The easiest way to do that is `DROP DATABASE ...` followed by `CREATE DATABASE
+...`. In the `psql` shell, first run the `\l` command to see all of the
+databases that are present:
+
+```shell
+cgawsbrokerprodyaobv93n2g3me5i=> \l
+                                                        List of databases
+              Name              |      Owner       | Encoding |   Collate   |    Ctype    |           Access privileges
+--------------------------------+------------------+----------+-------------+-------------+---------------------------------------
+ cgawsbrokerprodyaobv93n2g3me5i | ugsyn42g56vtykfr | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
+ postgres                       | ugsyn42g56vtykfr | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
+...
+```
+
+You will need the name of the database beginning with `cgawsbroker...` for the
+next step. To drop that database, you first have to connect to a different
+database (you can't drop the database that you are connected to). We connect to
+the default `postgres` database instead
+
+```shell
+cgawsbrokerprodyaobv93n2g3me5i=> \c postgres;
+psql (14.4, server 12.11)
+SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
+You are now connected to database "postgres" as user "ugsyn42g56vtykfr".
+```
+
+Now drop and create the database with the name above.
+
+```shell
+postgres=> DROP DATABASE cgawsbrokerprodyaobv93n2g3me5i;
+DROP DATABASE
+postgres=> CREATE DATABASE cgawsbrokerprodyaobv93n2g3me5i;
+CREATE DATABASE
+```
+
+Now the database is empty and Django will need to re-run all of its migrations
+in order for the app to start again.
+
+### Warnings
+
+This is a very intrusive procedure and it can go wrong in a number of ways.
+For example, if the running cloud.gov application goes down, the
+`connect-to-service` SSH tunnel will go away and if the app can't come back up
+(say, because the database has been dropped but not created) then it isn't
+possible to SSH back into the database to fix it and the Cloudfoundry
+resources may have to be completely deleted and re-created.

ops/scripts/deploy.sh

@@ -7,4 +7,5 @@
 cf target -o cisa-getgov-prototyping -s unstable
 cf push getgov-unstable -f ../ops/manifests/manifest-unstable.yaml
 
-cf run-task getgov-unstable --command 'python manage.py migrate' --name migrate
+# migrations need to be run manually. Developers can use this command
+#cf run-task getgov-unstable --command 'python manage.py migrate' --name migrate

src/run.sh

@@ -3,18 +3,6 @@
 set -o errexit
 set -o pipefail
 
-# Only run migrations on the zeroth index when in a cloud.gov environment
-if [[ -v CF_INSTANCE_INDEX && $CF_INSTANCE_INDEX == 0 ]]
-then
-  python manage.py migrate --settings=registrar.config.settings --noinput
-else
-  echo "Migrations did not run."
-  if [[ -v CF_INSTANCE_INDEX ]]
-  then
-    echo "CF Instance Index is ${CF_INSTANCE_INDEX}."
-  fi
-fi
-
 # Make sure that django's `collectstatic` has been run locally before pushing up to unstable,
 # so that the styles and static assets to show up correctly on unstable.