Merge remote-tracking branch 'origin/za/additional-data-transferred-domains' into za/additional-data-transferred-domains

docs/operations/data_migration.md

@@ -211,11 +211,12 @@ This will allow Docker to mount the files to a container (under `/app`) for our
 
 ## Transition Domains (Part 2) - Running the Migration Scripts
 
-*NOTE: While we recommend executing the following scripts individually (Steps 1-3), migrations can also be done 'all at once' using the "Run Migration Feature" in step 4.  Use with discretion.*
 
 ### STEP 1: Load Transition Domains
 
-Run the following command, making sure the file paths point to the right location.  This will parse the three given files and load the information into the TransitionDomain table. (NOTE: If working in cloud.gov, change "/app/tmp" to point to the `migrationdata/` directory)
+Run the following command, making sure the file paths point to the right location.  This will parse the three given files and load the information into the TransitionDomain table. 
+
+(NOTE: If working in cloud.gov, change "/app/tmp" to point to the `migrationdata/` directory and and remove "docker compose run -T app" from the command)
 ```shell
 docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt --debug
 ```
@@ -237,6 +238,8 @@ This will delete all the data in transtion_domain.  It is helpful if you want to
 Now that we've loaded all the data into TransitionDomain, we need to update the main Domain and DomainInvitation tables with this information.  
 In the same terminal as used in STEP 1, run the command below; 
 (This will parse the data in TransitionDomain and either create a corresponding Domain object, OR, if a corresponding Domain already exists, it will update that Domain with the incoming status. It will also create DomainInvitation objects for each user associated with the domain):
+
+(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
 ```shell
 docker compose run -T app ./manage.py transfer_transition_domains_to_domains --debug
 ```
@@ -251,9 +254,11 @@ Directs the script to load only the first 100 entries into the table.  You can a
 
 ### STEP 3: Send Domain invitations
 
-To send invitations for every transition domain in the transition domain table, execute the following command:
+To send invitation emails for every transition domain in the transition domain table, execute the following command:
+
+(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
 ```shell
-docker compose run -T app send_domain_invitations -s
+docker compose run -T app ./manage.py send_domain_invitations -s
 ```
 
 ### STEP 4: Test the results (Run the analyzer script)
@@ -263,6 +268,8 @@ This script's main function is to scan the transition domain and domain tables f
 #### OPTION 1 - ANALYZE ONLY
 
 To analyze our database without running migrations, execute the script without any optional arguments:
+
+(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
 ```shell
 docker compose run -T app ./manage.py master_domain_migrations --debug
 ```
@@ -270,6 +277,8 @@ docker compose run -T app ./manage.py master_domain_migrations --debug
 #### OPTION 2 - RUN MIGRATIONS FEATURE
 
 To run the migrations again (all above migration steps) before analyzing, execute the following command (read the documentation on the terminal arguments below.  Everything used by the migration scripts can also be passed into this script and will have the same effects).  NOTE: --debug and --prompt allow you to step through the migration process and exit it after each step if you need to.  It is recommended that you use these arguments when using the --runMigrations feature:
+
+(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
 ```shell
 docker compose run -T app ./manage.py master_domain_migrations --runMigrations --debug --prompt
 ```
@@ -278,13 +287,12 @@ docker compose run -T app ./manage.py master_domain_migrations --runMigrations -
 
 `--runMigrations`
 
-A boolean (default to true), which triggers running
-all scripts (in sequence) for transition domain migrations
+Runs all scripts (in sequence) for transition domain migrations
 
 `--migrationDirectory`
 
 The location of the files used for load_transition_domain migration script.
-(default is "migrationData" (This is the sandbox directory))
+(default is "migrationdata" (This is the sandbox directory))
 
 Example Usage:
 *--migrationDirectory /app/tmp*
@@ -310,13 +318,11 @@ Delimiter for the migration scripts to correctly parse the given text files.
 
 `--debug`
 
-A boolean (default to true), which activates additional print statements
+Activates additional print statements
 
 `--prompt`
 
-A boolean (default to true), which activates terminal prompts
-that allows the user to step through each portion of this
-script.
+Activates terminal prompts that allows the user to step through each portion of this script.
 
 `--limitParse`
 

docs/operations/runbooks/rotate_application_secrets.md

@@ -76,16 +76,24 @@ These are the client certificate and its private key used to identify the regist
 
 The private key is protected by a passphrase for safer transport and storage.
 
-These were generated with:
+These were generated with the following steps:
+
+### Step 1: Generate an unencrypted private key with a named curve
 
 ```bash
-openssl genpkey -out client.key \
-  -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
-  -aes-256-cbc
-openssl req -new -x509 -days 365 \
-  -key client.key -out client.crt \
-  -subj "/C=US/ST=DC/L=Washington/O=GSA/OU=18F/CN=GOV Prototype Registrar"
+openssl ecparam -name prime256v1 -genkey -out client_unencrypted.key
+```
 
+### Step 2: Create an encrypted private key with a passphrase
+
+```bash
+openssl pkcs8 -topk8 -v2 aes-256-cbc -in client_unencrypted.key -out client.key
+```
+
+### Generate the certificate
+
+```bash
+openssl req -new -x509 -days 365 -key client.key -out client.crt -subj "/C=US/ST=DC/L=Washington/O=GSA/OU=18F/CN=GOV Prototype Registrar"
 ```
 
 (If you can't use openssl on your computer directly, you can access it using Docker as `docker run --platform=linux/amd64 -it --rm -v $(pwd):/apps -w /apps alpine/openssl`.)
@@ -97,7 +105,14 @@ base64 client.key
 base64 client.crt
 ```
 
-You'll need to give the new certificate to the registry vendor _before_ rotating it in production.
+Note depending on your system you may need to instead run:
+
+```bash
+base64 -i client.key
+base64 -i client.crt
+```
+
+You'll need to give the new certificate to the registry vendor _before_ rotating it in production. Once it has been accepted by the vender, make sure to update the kdbx file on Google Drive.
 
 ## REGISTRY_HOSTNAME
 

src/migrationdata/README.md

@@ -1,8 +1,8 @@
 ## Purpose
-Use this folder for storing files for the migration process. Should otherwise be empty on local dev environments unless necessary. This folder must exist due to the nature of how data is stored on cloud.gov and the nature of the data we typically want to send. 
+Use this folder for storing files for the migration process. Should otherwise be empty on local dev environments unless necessary. This folder must exist due to the nature of how data is stored on cloud.gov and the nature of the data we want to send. 
 
 ## How do I migrate registrar data? 
 This process is detailed in [data_migration.md](../../docs/operations/data_migration.md)
 
 ## What kind of files can I store here?
-The intent is for PII data or otherwise, but this can exist in any format. Do note that the data contained in this file will be temporary, so after the app is restaged it will lose it (as long as nothing is committed). This is ideal for migration files as they write to our DB, but not for something you need to permanently hold onto.
+The intent is for PII data or otherwise, but this can exist in any format. Do note that the data contained in this file will be temporary, so after the app is restaged it will lose it. This is ideal for migration files as they write to our DB, but not for something you need to permanently hold onto.

src/registrar/assets/sass/_theme/_admin.scss

@@ -187,7 +187,7 @@ h1, h2, h3 {
 .object-tools li a,
 .object-tools p a {
     font-family: "Source Sans Pro Web", "Helvetica Neue", Helvetica, Roboto, Arial, sans-serif;
-    text-transform: capitalize!important;
+    text-transform: none!important;
     font-size: 14px!important;
 }
 

src/registrar/assets/sass/_theme/_alerts.scss

@@ -6,6 +6,10 @@
   }
 }
 
+.usa-alert__text.measure-none {
+  max-width: measure(none);
+}
+
 // The icon was off center for some reason
 // Fixes that issue
 @media (min-width: 64em){

src/registrar/config/settings.py

@@ -122,6 +122,8 @@ INSTALLED_APPS = [
     "api",
     # Only for generating documentation, uncomment to run manage.py generate_puml
     # "puml_generator",
+    # supports necessary headers for Django cross origin
+    "corsheaders",
 ]
 
 # Middleware are routines for processing web requests.
@@ -130,6 +132,8 @@ INSTALLED_APPS = [
 MIDDLEWARE = [
     # django-allow-cidr: enable use of CIDR IP ranges in ALLOWED_HOSTS
     "allow_cidr.middleware.AllowCIDRMiddleware",
+    # django-cors-headers: listen to cors responses
+    "corsheaders.middleware.CorsMiddleware",
     # serve static assets in production
     "whitenoise.middleware.WhiteNoiseMiddleware",
     # provide security enhancements to the request/response cycle
@@ -295,6 +299,11 @@ CSP_DEFAULT_SRC = allowed_sources
 CSP_FRAME_ANCESTORS = allowed_sources
 CSP_FORM_ACTION = allowed_sources
 
+# Cross-Origin Resource Sharing (CORS) configuration
+# Sets clients that allow access control to manage.get.gov
+# TODO: remove :8080 to see if we can have all localhost access
+CORS_ALLOWED_ORIGINS = ["http://localhost:8080", "https://beta.get.gov"]
+
 
 # Content-Length header is set by django.middleware.common.CommonMiddleware
 

src/registrar/management/commands/utility/extra_transition_domain_helper.py

@@ -632,8 +632,8 @@ class LoadExtraTransitionDomain:
 
 # TODO - change name
 @dataclass
-class PatternMap:
-    """Helper class that holds data and metadata about a requested file.
+class FileDataHolder:
+    """Helper class that holds data about a requested file.
 
     filename: str -> The desired filename to target. If no filename is given,
     it is assumed that you are passing in a filename pattern and it will look
@@ -823,7 +823,7 @@ class ExtraTransitionDomain:
             a list of values.
 
         return example:
-            EnumFilenames.AUTHORITY_ADHOC: PatternMap(
+            EnumFilenames.AUTHORITY_ADHOC: FileDataHolder(
                 authority_adhoc_filename,
                 self.strip_date_regex,
                 AuthorityAdhoc,
@@ -832,7 +832,7 @@ class ExtraTransitionDomain:
         """
         file_data = {}
         for file_type, filename, data_type, id_field in pattern_map_params:
-            file_data[file_type] = PatternMap(
+            file_data[file_type] = FileDataHolder(
                 filename,
                 self.strip_date_regex,
                 data_type,
@@ -896,7 +896,7 @@ class ExtraTransitionDomain:
 
     def clear_file_data(self):
         for item in self.file_data.values():
-            file_type: PatternMap = item
+            file_type: FileDataHolder = item
             file_type.data = {}
 
     def parse_csv_file(

src/registrar/templates/base.html

@@ -62,10 +62,11 @@
 
   {% if IS_DEMO_SITE %}
   <section aria-label="Alert" >
-      <div class="usa-alert usa-alert--warning usa-alert--no-icon">
+      <div class="usa-alert usa-alert--info">
       <div class="usa-alert__body">
-        <p class="usa-alert__text">
-        <strong>BETA SITE:</strong> We’re building a new way to get a .gov. Take a look around, but don’t rely on this site yet. This site is for testing purposes only. Don’t enter real data into any form on this site. To learn about requesting a .gov domain, visit <a href="https://get.gov" class="usa-link">get.gov</a>
+        <h4 class="usa-alert__heading">New domain requests are paused</h4>
+        <p class="usa-alert__text measure-none">
+        This is the new registrar for managing .gov domains. Note that we’re not accepting requests for new .gov domains until January 2024. Follow .gov updates at <a href="https://get.gov/updates/" class="usa-link">get.gov/updates/</a>.
         </p>
       </div>
     </div>

src/registrar/templates/django/admin/domain_change_list.html

@@ -15,7 +15,7 @@
     {% if has_add_permission %}
     <li>
         <a href="{% url 'admin:registrar_domain_add' %}" class="addlink">
-            Add Domain
+            Add domain
         </a>
     </li>
     {% endif %}

src/registrar/tests/data/test_migrationFilepaths.json

@@ -0,0 +1,12 @@
+{
+    "directory": "registrar/tests/data",
+    "agency_adhoc_filename": "test_agency_adhoc.txt",
+    "authority_adhoc_filename": "test_authority_adhoc.txt",
+    "organization_adhoc_filename": "test_organization_adhoc.txt",
+    "domain_adhoc_filename": "test_domain_types_adhoc.txt",
+    "domain_additional_filename": "test_domain_additional.txt",
+    "domain_contacts_filename": "test_domain_contacts.txt",
+    "domain_escrow_filename": "test_escrow_domains_daily.txt",
+    "domain_statuses_filename": "test_domain_statuses.txt",
+    "contacts_filename": "test_contacts.txt"
+}

src/registrar/tests/data/test_organization_adhoc.txt

@@ -1,6 +1,6 @@
 orgid|orgname|orgstreet|orgcity|orgstate|orgzip|orgcountrycode
 1|Flashdog|298 Monument Hill|Lakeland|Florida|33805|US
 2|Gigaclub|782 Mosinee Lane|Alexandria|Louisiana|71307|US
-3|Midel|376 Joe Pass|Waco|Texas|76705|US
+3|corrupt data|376 Joe Pass|Waco | corruption|Texas|76705|US
 4|Fanoodle|93001 Arizona Drive|Columbus|Ohio|43268|US
 5|Sushi|9999 Sushi Way|Columbus|Ohio|43268|US

src/registrar/tests/test_transition_domain_migrations.py

@@ -1,4 +1,4 @@
-from io import StringIO
+import datetime
 from django.test import TestCase
 
 from registrar.models import (
@@ -20,7 +20,7 @@ class TestMigrations(TestCase):
         # self.transfer_script = "transfer_transition_domains_to_domains",
         # self.master_script = "load_transition_domain",
 
-        self.test_data_file_location = "/app/registrar/tests/data"
+        self.test_data_file_location = "registrar/tests/data"
         self.test_domain_contact_filename = "test_domain_contacts.txt"
         self.test_contact_filename = "test_contacts.txt"
         self.test_domain_status_filename = "test_domain_statuses.txt"
@@ -32,6 +32,7 @@ class TestMigrations(TestCase):
         self.test_domain_types_adhoc = "test_domain_types_adhoc.txt"
         self.test_escrow_domains_daily = "test_escrow_domains_daily"
         self.test_organization_adhoc = "test_organization_adhoc.txt"
+        self.migration_json_filename = "test_migrationFilepaths.json"
 
     def tearDown(self):
         # Delete domain information
@@ -48,16 +49,8 @@ class TestMigrations(TestCase):
         with patch('registrar.management.commands.utility.terminal_helper.TerminalHelper.query_yes_no_exit', return_value=True):
             call_command(
                 "load_transition_domain",
-                f"{self.test_data_file_location}/{self.test_domain_contact_filename}",
-                f"{self.test_data_file_location}/{self.test_contact_filename}",
-                f"{self.test_data_file_location}/{self.test_domain_status_filename}",
-                directory=self.test_data_file_location,
-                agency_adhoc_filename=self.test_agency_adhoc_filename,
-                domain_additional_filename=self.test_domain_additional,
-                domain_escrow_filename=self.test_escrow_domains_daily,
-                domain_adhoc_filename=self.test_domain_types_adhoc,
-                organization_adhoc_filename=self.test_organization_adhoc,
-                authority_adhoc_filename=self.test_authority_adhoc_filename,
+                f"{self.test_data_file_location}/{self.migration_json_filename}",
+                directory=self.test_data_file_location
             )
 
     def run_transfer_domains(self):
@@ -68,12 +61,8 @@ class TestMigrations(TestCase):
             call_command(
                 "master_domain_migrations",
                 runMigrations=True,
-                migrationDirectory=f"{self.test_data_file_location}",
-                migrationFilenames=(
-                    f"{self.test_domain_contact_filename},"
-                    f"{self.test_contact_filename},"
-                    f"{self.test_domain_status_filename}"
-                ),
+                migrationDirectory=self.test_data_file_location,
+                migration_json_filename=self.migration_json_filename,
             )
 
     def compare_tables(
@@ -313,24 +302,26 @@ class TestMigrations(TestCase):
             expected_missing_domain_invitations,
         )
 
-        expected_anomaly_domains = Domain.objects.filter(name="anomaly.gov")
-        self.assertEqual(expected_anomaly_domains.count(), 1)
-        expected_anomaly = expected_anomaly_domains.get()
+        anomaly_domains = Domain.objects.filter(name="anomaly.gov")
+        self.assertEqual(anomaly_domains.count(), 1)
+        anomaly = anomaly_domains.get()
 
-        self.assertEqual(expected_anomaly.expiration_date, "test")
-        self.assertEqual(expected_anomaly.creation_date, "test")
-        self.assertEqual(expected_anomaly.name, "anomaly.gov")
-        self.assertEqual(expected_anomaly.state, "ready")
+        self.assertEqual(anomaly.expiration_date, datetime.date(2023, 3, 9))
+        self.assertEqual(
+            anomaly.created_at, datetime.datetime(2023, 11, 8, 17, 23, 46, 764663, tzinfo=datetime.timezone.utc)
+        )
+        self.assertEqual(anomaly.name, "anomaly.gov")
+        self.assertEqual(anomaly.state, "ready")
 
-        expected_testdomain_domains = Domain.objects.filter(name="anomaly.gov")
-        self.assertEqual(expected_testdomain_domains.count(), 1)
+        testdomain_domains = Domain.objects.filter(name="testdomain.gov")
+        self.assertEqual(testdomain_domains.count(), 1)
 
-        expected_testdomain = expected_testdomain_domains.get()
+        testdomain = testdomain_domains.get()
 
-        self.assertEqual(expected_testdomain.expiration_date, "test")
-        self.assertEqual(expected_testdomain.creation_date, "test")
-        self.assertEqual(expected_testdomain.name, "anomaly.gov")
-        self.assertEqual(expected_testdomain.state, "ready")
+        self.assertEqual(testdomain.expiration_date, datetime.date(2023, 3, 9))
+        self.assertEqual(testdomain.created_at, "test")
+        self.assertEqual(testdomain.name, "anomaly.gov")
+        self.assertEqual(testdomain.state, "ready")
 
         expected_domains = [
             Domain(

src/registrar/utility/csv_export.py

@@ -2,6 +2,8 @@ import csv
 from registrar.models.domain import Domain
 from registrar.models.domain_information import DomainInformation
 from registrar.models.public_contact import PublicContact
+from django.db.models import Value
+from django.db.models.functions import Coalesce
 
 
 def export_domains_to_writer(writer, columns, sort_fields, filter_condition):
@@ -61,7 +63,13 @@ def export_data_type_to_csv(csv_file):
         "Status",
         "Expiration Date",
     ]
-    sort_fields = ["domain__name"]
+    # Coalesce is used to replace federal_type of None with ZZZZZ
+    sort_fields = [
+        "organization_type",
+        Coalesce("federal_type", Value("ZZZZZ")),
+        "federal_agency",
+        "domain__name",
+    ]
     filter_condition = {
         "domain__state__in": [
             Domain.State.READY,
@@ -84,7 +92,13 @@ def export_data_full_to_csv(csv_file):
         "State",
         "Security Contact Email",
     ]
-    sort_fields = ["domain__name", "federal_agency", "organization_type"]
+    # Coalesce is used to replace federal_type of None with ZZZZZ
+    sort_fields = [
+        "organization_type",
+        Coalesce("federal_type", Value("ZZZZZ")),
+        "federal_agency",
+        "domain__name",
+    ]
     filter_condition = {
         "domain__state__in": [
             Domain.State.READY,
@@ -107,7 +121,13 @@ def export_data_federal_to_csv(csv_file):
         "State",
         "Security Contact Email",
     ]
-    sort_fields = ["domain__name", "federal_agency", "organization_type"]
+    # Coalesce is used to replace federal_type of None with ZZZZZ
+    sort_fields = [
+        "organization_type",
+        Coalesce("federal_type", Value("ZZZZZ")),
+        "federal_agency",
+        "domain__name",
+    ]
     filter_condition = {
         "organization_type__icontains": "federal",
         "domain__state__in": [