Merge branch 'main' of https://github.com/cisagov/manage.get.gov into rh/2258-update-ao-to-so

docs/architecture/decisions/0026-django-waffle-library.md

@@ -0,0 +1,47 @@
+# 26. Django Waffle library for Feature Flags
+
+Date: 2024-07-06
+
+## Status
+
+Approved
+
+## Context
+
+We release finished code twice weekly, allowing features to reach users quickly. However, several upcoming features require a series of changes that will need to be done over a few sprints and should only be displayed to users once we are all done. Thus, users would see half-finished features if we followed our standard process.
+
+At the same time, some of these features should only be turned on for users upon request (and likely during user research). We would want a way for our CISA users to turn this feature on and off for people without requiring a lengthy process or code changes.
+
+This brought us to finding solutions that could fix one or both of these problems.
+
+## Considered Options
+
+**Option 1:** Environment variables
+The environment allows developers to set a true or false value to the given variable, allowing implementation over multiple sprints when new features are encapsulated with this variable. The feature shows when the variable is on (true); otherwise, it remains hidden. Environment variables are also innate to Django, making them free to use; on top of that, we already use them for other things in our code.
+
+The downside is that you would need to go to cloud.gov or use the cf CLI to see the current settings on a sandbox. This is very technical, meaning only developers would really be able to see what features were set, and we would be the only ones able to adjust them. It would also be easy to accidentally have the feature on or off without noticing. This also would not solve the problem of turning features on and off quickly for a given user group.
+
+**Option 2:** Feature branches
+Like environment variables, using feature branches would be free and allow us to iterate on developing big features over multiple sprints. We would make a feature branch that developers working on that feature would push and pull from to iterate on. This quickly brings us to the downsides of this approach.
+
+Using feature branches, we do not solve the problem of being able to turn features on and off quickly for a user group. More importantly, by working in a separate branch for more than a sprint, we easily risk having out-of-sync migrations and merge conflicts that would slow development time and cause frustration. Out-of-sync migrations can also cause technical issues on sandboxes, further contributing to development frustration.
+
+**Option 3:** Feature flags
+Feature flags are free, allowing us to implement features over multiple sprints, and some libraries can apply features based on UserGroups while even more come with an interface for non-developers to control turning feature flags on and off. Going with this decision would also entail picking the correct library or product.
+
+**Option 3a:** Feature flags with Waffle
+The Waffle feature flag library is a highly recommended Django library for handling large features. It has clear documentation on turning on feature flags for user groups, which is one of the main problems it attempts to solve. It also provides "Samples" that can turn on flags for a certain percentage of users and "Switches" that can be used to turn features on and off holistically. The reviews from those who used it were highly favorable, some even mentioning how it beat out competitors like Gargoyl. It's also compatible with Django admin, providing a quick way to add the view of the flags in Django admin so any user with admin access can modify flags for their sandbox.
+
+The repo has had new releases every year since its the creation and looks to be well maintained, with many issues on the repo referring to new feature requests. 
+
+**Option 3b:** Feature flags with Gargoyl
+Gargoyl is another feature-flag library with Django, but it is no longer maintained, and reviews say it wasn't as easy to work with as Waffle. Using it would require forking the library, and many outstanding issues indicate bugs that need fixing. The mixed reviews from those who have done this and the less robust documentation were immediately huge cons to using this as an option.
+
+**Option 3c:** Paid feature flag system with GitHub integration- LaunchDarkly
+LaunchDarkly is a Fedramped solution with excellent reviews for controlling feature flags straight from GitHub to promote any team member easily controlling feature flags. However, the big con to this was that it would be a paid solution and would take time to procure, thus slowing down our ability to start on these significant features. We shouldn't consider LaunchDarkly because taking time to procure it would negatively affect our timeline, even if the budget was eventually approved.
+
+## Decision
+Option 3a, feature flags with the Django Waffle library
+
+## Consequences
+We are now reliant on the Waffle library for feature flags. As with any library, we would need to fork it if it ever became non-maintained with critical bugs. This doesn't seem likely in the near future, but if it occurred, we could complete the forking and fix any bug within a sprint without drastically impacting our timeline.

docs/developer/management_script_helpers.md

@@ -0,0 +1,65 @@
+# Terminal Helper Functions
+`terminal_helper.py` contains utility functions to assist with common terminal and script operations. 
+This file documents what they do and provides guidance on their usage.
+
+## TerminalColors
+`TerminalColors` provides ANSI color codes as variables to style terminal output.
+
+## ScriptDataHelper
+### bulk_update_fields
+
+`bulk_update_fields` performs a memory-efficient bulk update on a Django model in batches using a Paginator.
+
+## TerminalHelper
+### log_script_run_summary
+
+`log_script_run_summary` logs a summary of a script run, including counts of updated, skipped, and failed records.
+
+### print_conditional 
+
+`print_conditional` conditionally logs a statement at a specified severity if a condition is met.
+
+### prompt_for_execution
+
+`prompt_for_execution` prompts the user to inspect a string and confirm if they wish to proceed. Returns True if proceeding, False if skipping, or exits the script.
+
+### query_yes_no
+
+`query_yes_no` prompts the user with a yes/no question and returns True for "yes" or False for "no".
+
+### query_yes_no_exit
+
+`query_yes_no_exit` is similar to `query_yes_no` but includes an "exit" option to terminate the script.
+
+## PopulateScriptTemplate
+
+`PopulateScriptTemplate` is an abstract base class that provides a template for creating generic populate scripts. It handles logging and bulk updating for repetitive scripts that update a few fields. 
+
+### **Disclaimer** 
+This template is intended as a shorthand for simple scripts. It is not recommended for complex operations. See `transfer_federal_agency.py` for a straightforward example of how to use this template.
+
+### Step-by-step usage guide
+To create a script using `PopulateScriptTemplate`:
+1. Create a new class that inherits from `PopulateScriptTemplate`
+2. Implement the `update_record` method to define how each record should be updated
+3. Optionally, override the configuration variables and helper methods as needed
+4. Call `mass_update_records` within `handle` and run the script
+
+#### Template explanation
+
+The main method provided by `PopulateScriptTemplate` is `mass_update_records`. This method loops through each valid object (specified by `filter_conditions`) and updates the fields defined in `fields_to_update` using the `update_record` method.
+
+Before updating, `mass_update_records` prompts the user to confirm the proposed changes. If the user does not proceed, the script will exit.
+
+After processing the records, `mass_update_records` performs a bulk update on the specified fields using `ScriptDataHelper.bulk_update_fields` and logs a summary of the script run using `TerminalHelper.log_script_run_summary`.
+
+#### Config options
+The class provides the following optional configuration variables:
+- `prompt_title`: The header displayed by `prompt_for_execution` when the script starts (default: "Do you wish to proceed?")
+- `display_run_summary_items_as_str`: If True, runs `str(item)` on each item when printing the run summary for prettier output (default: False)
+- `run_summary_header`: The header for the script run summary printed after the script finishes (default: None)
+
+The class also provides helper methods:
+- `get_class_name`: Returns a display-friendly class name for the terminal prompt
+- `get_failure_message`: Returns the message to display if a record fails to update
+- `should_skip_record`: Defines the condition for skipping a record (by default, no records are skipped)

docs/operations/data_migration.md

@@ -698,6 +698,33 @@ Example: `cf ssh getgov-za`
 |:-:|:-------------------------- |:----------------------------------------------------------------------------|
 | 1 | **debug**                  | Increases logging detail. Defaults to False.                                |
 
+## Transfer federal agency script
+The transfer federal agency script adds the "federal_type" field on each associated DomainRequest, and uses that to populate the "federal_type" field on each FederalAgency.
+
+**Important:** When running this script, note that data generated by our fixtures will be inaccurate (since we assign random data to them). Use real data on this script.
+Do note that there is a check on record uniqueness. If two or more records do NOT have the same value for federal_type for any given federal agency, then the record is skipped. This protects against fixtures data when loaded with real data.
+
+### Running on sandboxes
+
+#### Step 1: Login to CloudFoundry
+```cf login -a api.fr.cloud.gov --sso```
+
+#### Step 2: SSH into your environment
+```cf ssh getgov-{space}```
+
+Example: `cf ssh getgov-za`
+
+#### Step 3: Create a shell instance
+```/tmp/lifecycle/shell```
+
+#### Step 4: Running the script
+```./manage.py transfer_federal_agency_type```
+
+### Running locally
+
+#### Step 1: Running the script
+```docker-compose exec app ./manage.py transfer_federal_agency_type```
+
 ## Email current metadata report
 
 ### Running on sandboxes