Merge pull request #726 from cisagov/pre-prod-sandbox

Add staging sandbox infrastructure

.github/ISSUE_TEMPLATE/developer-onboarding.md

@@ -83,6 +83,6 @@ export GPG_TTY
 
 ## Setting up developer sandbox
 
-We have two types of environments: stable, and sandbox. Stable gets deployed via tagged release every sprint, and developer sandboxes are given to get.gov developers to mess around in a production-like environment without disrupting stable. Each sandbox is namespaced and will automatically be deployed too when the appropriate branch syntax is used for that space in an open pull request. There are several things you need to setup to make the sandbox work for a developer. 
+We have three types of environments: stable, staging, and sandbox. Stable (production)and staging (pre-prod) get deployed via tagged release, and developer sandboxes are given to get.gov developers to mess around in a production-like environment without disrupting stable or staging. Each sandbox is namespaced and will automatically be deployed too when the appropriate branch syntax is used for that space in an open pull request. There are several things you need to setup to make the sandbox work for a developer. 
 
 All automation for setting up a developer sandbox is documented in the scripts for [creating a developer sandbox](../../ops/scripts/create_dev_sandbox.sh) and [removing a developer sandbox](../../ops/scripts/destroy_dev_sandbox.sh). A Cloud.gov organization administrator will have to perform the script in order to create the sandbox. 

.github/workflows/deploy-staging.yaml

@@ -0,0 +1,41 @@
+# This workflow runs on pushes of tagged commits.
+# "Releases" of tagged commits will deploy selected branch to staging.
+
+name: Build and deploy staging for tagged release
+
+on:
+  push:
+    paths-ignore:
+      - 'docs/**'
+      - '**.md'
+      - '.gitignore'
+      
+    tags:
+      - staging-*
+
+jobs:
+  deploy-staging:
+    if: ${{ github.ref_type == 'tag' }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Compile USWDS assets
+        working-directory: ./src
+        run: |
+          docker compose run node npm install && 
+          docker compose run node npx gulp copyAssets &&
+          docker compose run node npx gulp compile 
+      - name: Collect static assets 
+        working-directory: ./src
+        run: docker compose run app python manage.py collectstatic --no-input
+      - name: Deploy to cloud.gov sandbox
+        uses: 18f/cg-deploy-action@main
+        env:
+          DEPLOY_NOW: thanks
+        with:
+          cf_username: ${{ secrets.CF_STAGING_USERNAME }}
+          cf_password: ${{ secrets.CF_STAGING_PASSWORD }}
+          cf_org: cisa-getgov-prototyping
+          cf_space: staging
+          push_arguments: "-f ops/manifests/manifest-staging.yaml"

.github/workflows/migrate.yaml

@@ -14,6 +14,7 @@ on:
         description: Which environment should we run migrations for?
         options:
           - stable
+          - staging
           - gd
           - rb
           - ko

.github/workflows/reset-db.yaml

@@ -15,6 +15,7 @@ on:
         description: Which environment should we flush and re-load data for?
         options:
           - stable
+          - staging
           - gd
           - rb
           - ko

docs/developer/database-access.md

@@ -42,10 +42,9 @@ Optionally, load data from fixtures as well
 cf run-task getgov-ENVIRONMENT --wait --command 'python manage.py load' --name loaddata
 ```
 
-For the `stable` environment, developers don't have credentials so we need to
-run that command using Github Actions. Go to
+For the `stable` or `staging` environments, developers don't have credentials so we need to run that command using Github Actions. Go to
 <https://github.com/cisagov/getgov/actions/workflows/migrate.yaml> and select
-the "Run workflow" button, making sure that `stable` is selected.
+the "Run workflow" button, making sure that `stable` or `staging` depending on which envirornment you desire to update.
 
 ## Getting data for fixtures
 

docs/operations/README.md

@@ -35,7 +35,9 @@ Binding the database in `manifest-<ENVIRONMENT>.json` automatically inserts the
 
 # Deploy
 
-We have two types of environments: developer "sandboxes" and `stable`. Developers can deploy locally to their sandbox whenever they want. However, only our CD service can deploy to `stable`, and it does so when we make tagged releases of `main`. This is to ensure that we have a "golden" environment to point to, and can still test things out in a sandbox space. You should make sure all of the USWDS assets are compiled and collected before deploying to your sandbox. To deploy locally to `sandbox`:
+We have three types of environments: developer "sandboxes", `staging` and `stable`. Developers can deploy locally to their sandbox whenever they want. However, only our CD service can deploy to `staging` and `stable`, and it does so when we make tagged releases of `main`. For `staging`, this is done to ensure there is a non-production level test envirornment that can be used for user testing or for testing code before it is pushed to `stable`. `Staging` can be especially helpful when testing database changes or migrations that could have adververse affects in `stable`. On the other hand, `stable` is used to ensure that we have a "golden" environment to point to. We can refer to `stable` as our production environment and `staging` as our pre-production (pre-prod) environment. As such, code on main should always be tagged for `staging` before it is tagged for `stable`.
+
+You should make sure all of the USWDS assets are compiled and collected before deploying to your sandbox. To deploy locally to `sandbox`:
 
 For ease of use, you can run the `deploy.sh <sandbox name>` script in the `/src` directory to build the assets and deploy to your sandbox. Similarly, you could run `build.sh <sandbox name>` script to just compile and collect the assets without deploying.
 

ops/manifests/manifest-staging.yaml

@@ -0,0 +1,29 @@
+---
+applications:
+- name: getgov-staging
+  buildpacks:
+    - python_buildpack
+  path: ../../src
+  instances: 1
+  memory: 512M
+  stack: cflinuxfs4
+  timeout: 180
+  command: ./run.sh
+  health-check-type: http
+  health-check-http-endpoint: /health
+  env:
+    # Send stdout and stderr straight to the terminal without buffering
+    PYTHONUNBUFFERED: yup
+    # Tell Django where to find its configuration
+    DJANGO_SETTINGS_MODULE: registrar.config.settings
+    # Tell Django where it is being hosted
+    DJANGO_BASE_URL: https://getgov-staging.app.cloud.gov
+    # Tell Django how much stuff to log
+    DJANGO_LOG_LEVEL: INFO
+    # default public site location
+    GETGOV_PUBLIC_SITE_URL: https://beta.get.gov
+  routes:
+    - route: getgov-staging.app.cloud.gov
+  services:
+  - getgov-credentials
+  - getgov-staging-database

ops/scripts/create_dev_sandbox.sh

@@ -43,7 +43,7 @@ cp ops/scripts/manifest-sandbox-template.yaml ops/manifests/manifest-$1.yaml
 sed -i '' "s/ENVIRONMENT/$1/" "ops/manifests/manifest-$1.yaml"
 
 echo "Adding new environment to settings.py..."
-sed -i '' '/getgov-stable.app.cloud.gov/ {a\
+sed -i '' '/getgov-staging.app.cloud.gov/ {a\
     '\"getgov-$1.app.cloud.gov\"',
 }' src/registrar/config/settings.py
 
@@ -65,7 +65,7 @@ done
 echo "Creating new cloud.gov credentials for $1..."
 django_key=$(python3 -c 'from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())')
 openssl req -nodes -x509 -days 365 -newkey rsa:2048 -keyout private-$1.pem -out public-$1.crt
-login_key=$(base64 private-$1.pem)
+login_key=$(base64 -i private-$1.pem)
 jq -n --arg django_key "$django_key" --arg login_key "$login_key" '{"DJANGO_SECRET_KEY":$django_key,"DJANGO_SECRET_LOGIN_KEY":$login_key}' > credentials-$1.json
 cf cups getgov-credentials -p credentials-$1.json
 
@@ -105,11 +105,11 @@ echo
 echo "Moving on to setup Github automation..."
 
 echo "Adding new environment to Github Actions..."
-sed -i '' '/          - stable/ {a\
+sed -i '' '/          - staging/ {a\
           - '"$1"'
 }' .github/workflows/reset-db.yaml
 
-sed -i '' '/          - stable/ {a\
+sed -i '' '/          - staging/ {a\
           - '"$1"'
 }' .github/workflows/migrate.yaml
 

src/registrar/config/settings.py

@@ -564,6 +564,7 @@ SECURE_SSL_REDIRECT = True
 # web server configurations.
 ALLOWED_HOSTS = [
     "getgov-stable.app.cloud.gov",
+    "getgov-staging.app.cloud.gov",
     "getgov-gd.app.cloud.gov",
     "getgov-rb.app.cloud.gov",
     "getgov-ko.app.cloud.gov",