Merge branch 'main' into dk/1216-first-login-logic

docs/architecture/decisions/0024-production-release-cadence.md

@@ -0,0 +1,30 @@
+# 24. Production Release Cadence
+
+Date: 2023-11-02
+
+## Status
+
+In Review
+
+## Context
+
+Going into our first production launch we need a plan describing what our release cadence for both our staging and stable envirornments will be. Currently, we release to staging whenever there are significant changes made, but we have not been making releases to stable with the same frequency.
+
+## Considered Options
+
+**Option 1:** Releasing to stable/staging once a sprint
+Releasing once a sprint would mean that we release the past sprint's work to stable at the end of the current sprint. At the same point, the current sprint's work would be pushed to staging, thus making staging a full sprint ahead of stable. While this is more straight forward, it means our users would have to wait longer to see changes that weren't deemed critical.
+**Option 2:** Releasing to stable/staging once a week
+Releasing once a week would follow the same flow but with code being released to staging one week before the same code is released to stable. This would make stable only one week behind staging and would allow us to roll out minor bug fixes and faster with greater speed. The negative side is that we have less time to see if errors occur on staging
+
+In both of the above scenarios the release date would fall on the same day of the week that the sprint starts, which is currently a Wednesday. Additionally, in both scenarios the release commits would eventually be tagged with both a staging and stable tag. Furthermore, critical bugs or features would be exempt from these restrictions based on the product owner's discretion.
+
+## Decision
+
+We decided to go with option 2 and release once a week once in production. This will allow us to give users features and bug fixes faster while still allowing enough time on staging for quality to be maintained.
+
+## Consequences
+
+Work not completed by end of the sprint will have to wait to be added to stable. Also, making quick fixes for bugs that are found on stable will be a little more complicated to fix.
+
+When first going into production, staging and stable will start with the same code base. The following week a new release will be made to staging, but not stable as no code will have been on staging long enough to warrant another release. Thus just at the start of launch stable will be essentially frozen for 2 weeks, not one.

docs/operations/README.md

@@ -43,6 +43,21 @@ For ease of use, you can run the `deploy.sh <sandbox name>` script in the `/src`
 
 Your sandbox space should've been setup as part of the onboarding process. If this was not the case, please have an admin follow the instructions [here](../../.github/ISSUE_TEMPLATE/developer-onboarding.md#setting-up-developer-sandbox).
 
+## Stable and Staging Release Rules
+
+Releases will be made for staging and stable every week starting on the first day of the sprint (Wednesday), with the second release of the sprint occuring halfway through the sprint. With the exception of first time going into production, these releases will NOT have the same code. The release to stable will be the same commit that was tagged for staging one week prior, making stable one week behind staging. Further, this means staging can be up to a week behind the main branch of code.
+
+If a bug fix or feature needs to be made to stable out of the normal cycle, this can only be done at the product owner's request.
+
+## Making bug fixes on stable during production
+
+In the case where a bug fix or feature needs to be added outside of the normal cycle, the code-fix branch and release will be handled differently than normal:
+
+1. Code will need to be branched NOT off of main, but off of the same commit as the most recent stable commit. This should be the one tagged with the most recent vX.XX.XX value.
+2. After making the bug fix, the approved PR will branch will be tagged with a new release tag, incrementing the patch value from the last commit number.
+3. This branch then needs to be merged to main per the usual process.
+4. This same branch should be merged into staging.
+
 ## Serving static assets
 We are using [WhiteNoise](http://whitenoise.evans.io/en/stable/index.html) plugin to serve our static assets on cloud.gov. This plugin is added to the `MIDDLEWARE` list in our apps `settings.py`.
 
@@ -159,3 +174,27 @@ it with the latest model schema. Once launched, this should never be used on
 the `stable` environment, but during development, it may be useful on the
 various sandbox environments. After launch, some schema changes may take the
 involvement of a skilled DBA to fix problems like this.
+
+# Bug triage
+
+Bugs on production software need to be documented quickly and triaged to determine if fixes need to be made outside of the normal release cadence. Triage levels will be Critical, High, Medium, and Low to indicate the level of priority for fix, not neccessarily the level of severity. See below for more details
+
+**Critical**- should only be determined by the product owner and means the fix for this critical bug needs to have a quick fix for it created ASAP. This is the only case where a bug fix can be added outside of the normal release cycle and directly onto the stable release.
+**High**- Can be determined by product owner or other team member, and indicates this bug is critical enough to warrant being added into the current sprint.
+**Medium**- Should be added to a sprint coming up but is not blocking users, or enough users to warrant rushing it into a sprint
+**Low**- A minor bug, that could even wait until after the next big launch date to be implemented.
+
+## Steps for Triaging
+
+1. When a bug is found, whether by a developer/designer or from feedback from an end user, a ticket should be made immediately. The actual maker of the ticket can be a member of the product team as needed.
+2. This bug ticket immediately gets a priority added Critical/High/Medium/Low, with Critical requiring the product owner's consent.
+3. Anything marked as `critical` should be refined immediately and engineering should be notified in our Slack dev channel that a Critical ticket has been created (if not already notified)
+4. All items not marked as `critical` by the product owner can wait until refinement to be refined and may have their prioirty level changed during that meeting.
+
+## Steps for dealing with Critical Bugs
+
+1. Once the critical bug ticket is refined and the bug is clear, an engineer should be assigned to work on it. (No ticket, no work)
+2. At the same point, two other engineers should be assigned to review the PR once it's made. One of the reviewing engineers can be subsititued for a designer if this is a design/content/other user facing bug fix.
+3. In the case where the engineering lead is is unresponsive or unavailable to assign the ticket immediately, the product team will make sure an engineer volunteers or is assigned to the ticket/PR review ASAP.
+4. Once done, the developer must make a PR and should tag the assigned PR reviewers in our Slack dev channel stating that the PR is now waiting on their review. These reviewers should drop other tasks in order to review this promptly.
+5. See the the section above on [Making bug fixes on stable](#making-bug-fixes-on-stable-during-production) for how to push changes to stable once the PR is approved