diff --git a/docs/operations/data_migration.md b/docs/operations/data_migration.md index 00f84a897..316098378 100644 --- a/docs/operations/data_migration.md +++ b/docs/operations/data_migration.md @@ -87,12 +87,12 @@ FILE 3: **escrow_domain_statuses.daily.gov.GOV.txt** -> has the map of domains a We need to run a few scripts to parse these files into our domain tables. We can do this both locally and in a sandbox. -## OPTION 1: SANDBOX -## Load migration data onto a production or sandbox environment +### SANDBOX MIGRATION SETUP +### Load migration data onto a production or sandbox environment **WARNING:** All files uploaded in this manner are temporary, i.e. they will be deleted when the app is restaged. Do not use this method to store data you want to keep around permanently. -### STEP 1: Use scp to transfer data +#### STEP 1: Use scp to transfer data CloudFoundry supports scp as means of transferring data locally to our environment. If you are dealing with a batch of files, try sending across a tar.gz and unpacking that. **Login to Cloud.gov** @@ -134,7 +134,7 @@ Copy this code into the password prompt from earlier. NOTE: You can use different utilities to copy this onto the clipboard for you. If you are on Windows, try the command `cf ssh-code | clip`. On Mac, this will be `cf ssh-code | pbcopy` -### STEP 2: Transfer uploaded files to the getgov directory +#### STEP 2: Transfer uploaded files to the getgov directory Due to the nature of how Cloud.gov operates, the getgov directory is dynamically generated whenever the app is built under the tmp/ folder. We can directly upload files to the tmp/ folder but cannot target the generated getgov folder directly, as we need to spin up a shell to access this. From here, we can move those uploaded files into the getgov directory using the `cat` command. Note that you will have to repeat this for each file you want to move, so it is better to use a tar.gz for multiple, and unpack it inside of the `datamigration` folder. **SSH into your sandbox** @@ -168,7 +168,7 @@ tar -xvf migrationdata/{FILE_NAME}.tar.gz -C migrationdata/ --strip-components=1 -#### Manual method +##### Manual method If the `cat_files_into_getgov.py` script isn't working, follow these steps instead. **Move the desired file into the correct directory** @@ -178,34 +178,37 @@ cat ../tmp/{filename} > migrationdata/{filename} ``` -### STEP 3: Load Transition Domain data into TransitionDomain table -Run the following script to transfer the existing data on our .txt files to our DB. -```shell -./manage.py load_transition_domain migrationdata/escrow_domain_contacts.daily.gov.GOV.txt migrationdata/escrow_contacts.daily.gov.GOV.txt migrationdata/escrow_domain_statuses.daily.gov.GOV.txt +#### You are now ready to run migration scripts (see "Running the Migration Scripts") ``` -## OPTION 2: LOCAL -## Load migration data onto our local environments +``` +### LOCAL MIGRATION SETUP (TESTING PURPOSES ONLY) +### Load migration data onto our local environments -Transferring this data from these files into our domain tables happens in two steps; +***IMPORTANT: only use test data, to avoid publicizing PII in our public repo.*** -***IMPORTANT: only run the following locally, to avoid publicizing PII in our public repo.*** - -### STEP 1: Load Transition Domain data into TransitionDomain table - -**SETUP** -In order to use the management command, we need to add the files to a folder under `src/`. +In order to run the scripts locally, we need to add the files to a folder under `src/`. This will allow Docker to mount the files to a container (under `/app`) for our use. - Create a folder called `tmp` underneath `src/` - Add the above files to this folder - Open a terminal and navigate to `src/` -Then run the following command (This will parse the three files in your `tmp` folder and load the information into the TransitionDomain table); +#### You are now ready to run migration scripts (see "Running the Migration Scripts") + +### RUNNING THE MIGRATION SCRIPTS + +**NOTE: Whil we recommend executing the following scripts individually (Steps 1-3), migrations can also be done 'all at once' using the "Run Migration Feature" in step 4. Use with discretion.** + +#### 1 - Load Transition Domains +Run the following command (This will parse the three files in your `tmp` folder and load the information into the TransitionDomain table); ```shell -docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt +docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt --debug ``` +**For Sandbox**: +Change "/app/tmp" to point to the sandbox directory + **OPTIONAL COMMAND LINE ARGUMENTS**: `--debug` This will print out additional, detailed logs. @@ -214,7 +217,7 @@ This will print out additional, detailed logs. Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes. `--resetTable` -This will delete all the data loaded into transtion_domain. It is helpful if you want to see the entries reload from scratch or for clearing test data. +This will delete all the data in transtion_domain. It is helpful if you want to see the entries reload from scratch or for clearing test data. ### STEP 2: Transfer Transition Domain data into main Domain tables @@ -224,7 +227,7 @@ Now that we've loaded all the data into TransitionDomain, we need to update the In the same terminal as used in STEP 1, run the command below; (This will parse the data in TransitionDomain and either create a corresponding Domain object, OR, if a corresponding Domain already exists, it will update that Domain with the incoming status. It will also create DomainInvitation objects for each user associated with the domain): ```shell -docker compose run -T app ./manage.py transfer_transition_domains_to_domains +docker compose run -T app ./manage.py transfer_transition_domains_to_domains --debug ``` **OPTIONAL COMMAND LINE ARGUMENTS**: @@ -232,4 +235,68 @@ docker compose run -T app ./manage.py transfer_transition_domains_to_domains This will print out additional, detailed logs. `--limitParse 100` -Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes. \ No newline at end of file +Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes. + +### STEP 3: Send Domain invitations +### Run the send invitations script + +To send invitations for every transition domain in the transition domain table, execute the following command: +`docker compose run -T app send_domain_invitations -s` + +### STEP 4: Test the results +### Run the migration analyzer + +This script's main function is to scan the transition domain and domain tables for any anomalies. It produces a simple report of missing or duplicate data. NOTE: some missing data might be expected depending on the nature of our migrations so use best judgement when evaluating the results. + +**ANALYZE ONLY** +To analyze our database without running migrations, execute the script without any optional arguments: +`docker compose run -T app ./manage.py master_domain_migrations --debug` + +**RUN MIGRATIONS FEATURE** +To run the migrations again (all above migration steps) before analyzing, execute the following command (read the documentation on the terminal arguments below. Everything used by the migration scripts can also be passed into this script and will have the same effects). NOTE: --debug and --prompt allow you to step through the migration process and exit it after each step if you need to. It is recommended that you use these arguments when using the --runMigrations feature: +`docker compose run -T app ./manage.py master_domain_migrations --runMigrations --debug --prompt` + +#### OPTIONAL ARGUMENTS +`--runMigrations` +A boolean (default to true), which triggers running +all scripts (in sequence) for transition domain migrations + +`--migrationDirectory` +**default="migrationData"** (<--This is the sandbox directory) +The location of the files used for load_transition_domain migration script +EXAMPLE USAGE: +--migrationDirectory /app/tmp + +`--migrationFilenames` +**default=escrow_domain_contacts.daily.gov.GOV.txt,escrow_contacts.daily.gov.GOV.txt,escrow_domain_statuses.daily.gov.GOV.txt** (<--These are the usual names for the files. The script will throw a warning if it cannot find these exact files, in which case you will need to supply the correct filenames) +The files used for load_transition_domain migration script. +Must appear IN ORDER and comma-delimited: +EXAMPLE USAGE: +--migrationFilenames domain_contacts_filename.txt,contacts_filename.txt,domain_statuses_filename.txt +where... +- domain_contacts_filename is the Data file with domain contact information +- contacts_filename is the Data file with contact information +- domain_statuses_filename is the Data file with domain status information + +`--sep` +Delimiter for the migration scripts to correctly parse the given text files. +(usually this can remain at default value of |) + +`--debug` +A boolean (default to true), which activates additional print statements + +`--prompt` +A boolean (default to true), which activates terminal prompts +that allows the user to step through each portion of this +script. + +`--limitParse` +Used by the migration scripts (load_transition_domain) to set the limit for the +number of data entries to insert. Set to 0 (or just don't use this +argument) to parse every entry. This was provided primarily for testing +purposes + +`--resetTable` +Used by the migration scripts to trigger a prompt for deleting all table entries. +Useful for testing purposes, but USE WITH CAUTION +``` \ No newline at end of file diff --git a/src/epplibwrapper/__init__.py b/src/epplibwrapper/__init__.py index d0138d73c..dd6664a3a 100644 --- a/src/epplibwrapper/__init__.py +++ b/src/epplibwrapper/__init__.py @@ -45,7 +45,7 @@ except NameError: # Attn: these imports should NOT be at the top of the file try: from .client import CLIENT, commands - from .errors import RegistryError, ErrorCode, CANNOT_CONTACT_REGISTRY, GENERIC_ERROR + from .errors import RegistryError, ErrorCode from epplib.models import common, info from epplib.responses import extensions from epplib import responses @@ -61,6 +61,4 @@ __all__ = [ "info", "ErrorCode", "RegistryError", - "CANNOT_CONTACT_REGISTRY", - "GENERIC_ERROR", ] diff --git a/src/epplibwrapper/errors.py b/src/epplibwrapper/errors.py index dba5f328c..d34ed5e91 100644 --- a/src/epplibwrapper/errors.py +++ b/src/epplibwrapper/errors.py @@ -1,8 +1,5 @@ from enum import IntEnum -CANNOT_CONTACT_REGISTRY = "Update failed. Cannot contact the registry." -GENERIC_ERROR = "Value entered was wrong." - class ErrorCode(IntEnum): """ diff --git a/src/registrar/assets/js/get-gov.js b/src/registrar/assets/js/get-gov.js index 851de8fcf..1c678a4d6 100644 --- a/src/registrar/assets/js/get-gov.js +++ b/src/registrar/assets/js/get-gov.js @@ -231,41 +231,15 @@ function handleValidationClick(e) { /** - * An IIFE that attaches a click handler for our dynamic nameservers form - * - * Only does something on a single page, but it should be fast enough to run - * it everywhere. + * Prepare the namerservers and DS data forms delete buttons + * We will call this on the forms init, and also every time we add a form + * */ -(function prepareNameserverForms() { - let serverForm = document.querySelectorAll(".server-form"); - let container = document.querySelector("#form-container"); - let addButton = document.querySelector("#add-nameserver-form"); - let totalForms = document.querySelector("#id_form-TOTAL_FORMS"); - - let formNum = serverForm.length-1; - if (addButton) - addButton.addEventListener('click', addForm); - - function addForm(e){ - let newForm = serverForm[2].cloneNode(true); - let formNumberRegex = RegExp(`form-(\\d){1}-`,'g'); - let formLabelRegex = RegExp(`Name server (\\d){1}`, 'g'); - let formExampleRegex = RegExp(`ns(\\d){1}`, 'g'); - - formNum++; - newForm.innerHTML = newForm.innerHTML.replace(formNumberRegex, `form-${formNum}-`); - newForm.innerHTML = newForm.innerHTML.replace(formLabelRegex, `Name server ${formNum+1}`); - newForm.innerHTML = newForm.innerHTML.replace(formExampleRegex, `ns${formNum+1}`); - container.insertBefore(newForm, addButton); - newForm.querySelector("input").value = ""; - - totalForms.setAttribute('value', `${formNum+1}`); - } -})(); - -function prepareDeleteButtons() { +function prepareDeleteButtons(formLabel) { let deleteButtons = document.querySelectorAll(".delete-record"); let totalForms = document.querySelector("#id_form-TOTAL_FORMS"); + let isNameserversForm = document.title.includes("DNS name servers |"); + let addButton = document.querySelector("#add-form"); // Loop through each delete button and attach the click event listener deleteButtons.forEach((deleteButton) => { @@ -273,13 +247,15 @@ function prepareDeleteButtons() { }); function removeForm(e){ - let formToRemove = e.target.closest(".ds-record"); + let formToRemove = e.target.closest(".repeatable-form"); formToRemove.remove(); - let forms = document.querySelectorAll(".ds-record"); + let forms = document.querySelectorAll(".repeatable-form"); totalForms.setAttribute('value', `${forms.length}`); let formNumberRegex = RegExp(`form-(\\d){1}-`, 'g'); - let formLabelRegex = RegExp(`DS Data record (\\d){1}`, 'g'); + let formLabelRegex = RegExp(`${formLabel} (\\d+){1}`, 'g'); + // For the example on Nameservers + let formExampleRegex = RegExp(`ns(\\d+){1}`, 'g'); forms.forEach((form, index) => { // Iterate over child nodes of the current element @@ -294,48 +270,88 @@ function prepareDeleteButtons() { }); }); - Array.from(form.querySelectorAll('h2, legend')).forEach((node) => { - node.textContent = node.textContent.replace(formLabelRegex, `DS Data record ${index + 1}`); + // h2 and legend for DS form, label for nameservers + Array.from(form.querySelectorAll('h2, legend, label, p')).forEach((node) => { + + // Ticket: 1192 + // if (isNameserversForm && index <= 1 && !node.innerHTML.includes('*')) { + // // Create a new element + // const newElement = document.createElement('abbr'); + // newElement.textContent = '*'; + // // TODO: finish building abbr + + // // Append the new element to the parent + // node.appendChild(newElement); + // // Find the next sibling that is an input element + // let nextInputElement = node.nextElementSibling; + + // while (nextInputElement) { + // if (nextInputElement.tagName === 'INPUT') { + // // Found the next input element + // console.log(nextInputElement); + // break; + // } + // nextInputElement = nextInputElement.nextElementSibling; + // } + // nextInputElement.required = true; + // } + + // Ticket: 1192 - remove if + if (!(isNameserversForm && index <= 1)) { + node.textContent = node.textContent.replace(formLabelRegex, `${formLabel} ${index + 1}`); + node.textContent = node.textContent.replace(formExampleRegex, `ns${index + 1}`); + } }); + + // Display the add more button if we have less than 13 forms + if (isNameserversForm && forms.length <= 13) { + addButton.classList.remove("display-none") + } }); } } /** - * An IIFE that attaches a click handler for our dynamic DNSSEC forms + * An IIFE that attaches a click handler for our dynamic formsets * + * Only does something on a few pages, but it should be fast enough to run + * it everywhere. */ -(function prepareDNSSECForms() { - let serverForm = document.querySelectorAll(".ds-record"); +(function prepareFormsetsForms() { + let repeatableForm = document.querySelectorAll(".repeatable-form"); let container = document.querySelector("#form-container"); - let addButton = document.querySelector("#add-ds-form"); + let addButton = document.querySelector("#add-form"); let totalForms = document.querySelector("#id_form-TOTAL_FORMS"); + let cloneIndex = 0; + let formLabel = ''; + let isNameserversForm = document.title.includes("DNS name servers |"); + if (isNameserversForm) { + cloneIndex = 2; + formLabel = "Name server"; + } else if ((document.title.includes("DS Data |")) || (document.title.includes("Key Data |"))) { + formLabel = "DS Data record"; + } // Attach click event listener on the delete buttons of the existing forms - prepareDeleteButtons(); + prepareDeleteButtons(formLabel); - // Attack click event listener on the add button if (addButton) addButton.addEventListener('click', addForm); - /* - * Add a formset to the end of the form. - * For each element in the added formset, name the elements with the prefix, - * form-{#}-{element_name} where # is the index of the formset and element_name - * is the element's name. - * Additionally, update the form element's metadata, including totalForms' value. - */ function addForm(e){ - let forms = document.querySelectorAll(".ds-record"); + let forms = document.querySelectorAll(".repeatable-form"); let formNum = forms.length; - let newForm = serverForm[0].cloneNode(true); + let newForm = repeatableForm[cloneIndex].cloneNode(true); let formNumberRegex = RegExp(`form-(\\d){1}-`,'g'); - let formLabelRegex = RegExp(`DS Data record (\\d){1}`, 'g'); + let formLabelRegex = RegExp(`${formLabel} (\\d){1}`, 'g'); + // For the eample on Nameservers + let formExampleRegex = RegExp(`ns(\\d){1}`, 'g'); formNum++; newForm.innerHTML = newForm.innerHTML.replace(formNumberRegex, `form-${formNum-1}-`); - newForm.innerHTML = newForm.innerHTML.replace(formLabelRegex, `DS Data record ${formNum}`); + newForm.innerHTML = newForm.innerHTML.replace(formLabelRegex, `${formLabel} ${formNum}`); + newForm.innerHTML = newForm.innerHTML.replace(formExampleRegex, `ns${formNum}`); container.insertBefore(newForm, addButton); let inputs = newForm.querySelectorAll("input"); @@ -379,9 +395,13 @@ function prepareDeleteButtons() { totalForms.setAttribute('value', `${formNum}`); // Attach click event listener on the delete buttons of the new form - prepareDeleteButtons(); - } + prepareDeleteButtons(formLabel); + // Hide the add more button if we have 13 forms + if (isNameserversForm && formNum == 13) { + addButton.classList.add("display-none") + } + } })(); /** diff --git a/src/registrar/assets/sass/_theme/_forms.scss b/src/registrar/assets/sass/_theme/_forms.scss index 38b42c3d0..d0bfbee67 100644 --- a/src/registrar/assets/sass/_theme/_forms.scss +++ b/src/registrar/assets/sass/_theme/_forms.scss @@ -4,6 +4,10 @@ margin-top: units(3); } +.usa-form .usa-button.margin-bottom-075 { + margin-bottom: units(1.5); +} + .usa-form .usa-button.margin-top-1 { margin-top: units(1); } diff --git a/src/registrar/forms/domain.py b/src/registrar/forms/domain.py index 6bbade5ef..3aca7af6d 100644 --- a/src/registrar/forms/domain.py +++ b/src/registrar/forms/domain.py @@ -5,8 +5,12 @@ from django.core.validators import MinValueValidator, MaxValueValidator, RegexVa from django.forms import formset_factory from phonenumber_field.widgets import RegionalPhoneNumberWidget +from registrar.utility.errors import ( + NameserverError, + NameserverErrorCodes as nsErrorCodes, +) -from ..models import Contact, DomainInformation +from ..models import Contact, DomainInformation, Domain from .common import ( ALGORITHM_CHOICES, DIGEST_TYPE_CHOICES, @@ -19,16 +23,78 @@ class DomainAddUserForm(forms.Form): email = forms.EmailField(label="Email") +class IPAddressField(forms.CharField): + def validate(self, value): + super().validate(value) # Run the default CharField validation + + class DomainNameserverForm(forms.Form): """Form for changing nameservers.""" + domain = forms.CharField(widget=forms.HiddenInput, required=False) + server = forms.CharField(label="Name server", strip=True) - # when adding IPs to this form ensure they are stripped as well + + ip = forms.CharField(label="IP Address (IPv4 or IPv6)", strip=True, required=False) + + def clean(self): + # clean is called from clean_forms, which is called from is_valid + # after clean_fields. it is used to determine form level errors. + # is_valid is typically called from view during a post + cleaned_data = super().clean() + self.clean_empty_strings(cleaned_data) + server = cleaned_data.get("server", "") + ip = cleaned_data.get("ip", None) + # remove ANY spaces in the ip field + ip = ip.replace(" ", "") + domain = cleaned_data.get("domain", "") + + ip_list = self.extract_ip_list(ip) + + if ip and not server and ip_list: + self.add_error("server", NameserverError(code=nsErrorCodes.MISSING_HOST)) + elif server: + self.validate_nameserver_ip_combo(domain, server, ip_list) + + return cleaned_data + + def clean_empty_strings(self, cleaned_data): + ip = cleaned_data.get("ip", "") + if ip and len(ip.strip()) == 0: + cleaned_data["ip"] = None + + def extract_ip_list(self, ip): + return [ip.strip() for ip in ip.split(",")] if ip else [] + + def validate_nameserver_ip_combo(self, domain, server, ip_list): + try: + Domain.checkHostIPCombo(domain, server, ip_list) + except NameserverError as e: + if e.code == nsErrorCodes.GLUE_RECORD_NOT_ALLOWED: + self.add_error( + "server", + NameserverError( + code=nsErrorCodes.GLUE_RECORD_NOT_ALLOWED, + nameserver=domain, + ip=ip_list, + ), + ) + elif e.code == nsErrorCodes.MISSING_IP: + self.add_error( + "ip", + NameserverError( + code=nsErrorCodes.MISSING_IP, nameserver=domain, ip=ip_list + ), + ) + else: + self.add_error("ip", str(e)) NameserverFormset = formset_factory( DomainNameserverForm, extra=1, + max_num=13, + validate_max=True, ) diff --git a/src/registrar/management/commands/cat_files_into_getgov.py b/src/registrar/management/commands/cat_files_into_getgov.py index 5de44ba73..6c46994ea 100644 --- a/src/registrar/management/commands/cat_files_into_getgov.py +++ b/src/registrar/management/commands/cat_files_into_getgov.py @@ -3,9 +3,12 @@ import glob import logging import os +import string from django.core.management import BaseCommand +from registrar.management.commands.utility.terminal_helper import TerminalHelper + logger = logging.getLogger(__name__) @@ -27,6 +30,7 @@ class Command(BaseCommand): def handle(self, **options): file_extension: str = options.get("file_extension").lstrip(".") directory = options.get("directory") + helper = TerminalHelper() # file_extension is always coerced as str, Truthy is OK to use here. if not file_extension or not isinstance(file_extension, str): @@ -38,29 +42,59 @@ class Command(BaseCommand): for src_file_path in matching_extensions: filename = os.path.basename(src_file_path) + exit_status = -1 do_command = True - exit_status: int desired_file_path = f"{directory}/{filename}" if os.path.exists(desired_file_path): # For linter - prompt = " Do you want to replace it? (y/n) " - replace = input(f"{desired_file_path} already exists. {prompt}") - if replace.lower() != "y": + prompt = "Do you want to replace it?" + replace = f"{desired_file_path} already exists. {prompt}" + if not helper.query_yes_no(replace): do_command = False - if do_command: - copy_from = f"../tmp/{filename}" - self.cat(copy_from, desired_file_path) - exit_status = os.system(f"cat ../tmp/{filename} > {desired_file_path}") - - if exit_status == 0: - logger.info(f"Successfully copied {filename}") - else: - logger.error(f"Failed to copy {filename}") + try: + if do_command: + copy_from = f"../tmp/{filename}" + exit_status = self.cat(copy_from, desired_file_path) + except ValueError as err: + raise err + finally: + if exit_status == 0: + logger.info(f"Successfully copied {filename}") + else: + logger.error(f"Failed to copy {filename}") def cat(self, copy_from, copy_to): """Runs the cat command to copy_from a location to copy_to a location""" - exit_status = os.system(f"cat {copy_from} > {copy_to}") + + # copy_from will be system defined + self.check_file_path(copy_from, check_directory=False) + self.check_file_path(copy_to) + + # This command can only be ran from inside cf ssh getgov-{sandbox} + # It has no utility when running locally, and to exploit this + # you would have to have ssh access anyway, which is a bigger problem. + exit_status = os.system(f"cat {copy_from} > {copy_to}") # nosec return exit_status + + def check_file_path(self, file_path: str, check_directory=True): + """Does a check on user input to ensure validity""" + if not isinstance(file_path, str): + raise ValueError("Invalid path provided") + + # Remove any initial/final whitespace + file_path = file_path.strip() + + # Check for any attempts to move up in the directory structure + if ".." in file_path and check_directory: + raise ValueError("Moving up in the directory structure is not allowed") + + # Check for any invalid characters + valid_chars = f"/-_.() {string.ascii_letters}{string.digits}" + for char in file_path: + if char not in valid_chars: + raise ValueError(f"Invalid character {char} in file path") + + return file_path diff --git a/src/registrar/management/commands/master_domain_migrations.py b/src/registrar/management/commands/master_domain_migrations.py index a5660b704..206b63276 100644 --- a/src/registrar/management/commands/master_domain_migrations.py +++ b/src/registrar/management/commands/master_domain_migrations.py @@ -8,11 +8,6 @@ import logging import argparse import sys -import os - -from django.test import Client - -from django_fsm import TransitionNotAllowed # type: ignore from django.core.management import BaseCommand from django.core.management import call_command @@ -22,7 +17,6 @@ from registrar.models import ( DomainInformation, DomainInvitation, TransitionDomain, - User, ) from registrar.management.commands.utility.terminal_helper import ( @@ -100,19 +94,26 @@ class Command(BaseCommand): parser.add_argument( "--migrationDirectory", default="migrationData", - help="The location of the files used for load_transition_domain migration script", + help=( + "The location of the files used for" + "load_transition_domain migration script" + ), ) parser.add_argument( "--migrationFilenames", - default="escrow_domain_contacts.daily.gov.GOV.txt,escrow_contacts.daily.gov.GOV.txt,escrow_domain_statuses.daily.gov.GOV.txt", - help="""The files used for load_transition_domain migration script. - Must appear IN ORDER and separated by commas: + default="""escrow_domain_contacts.daily.gov.GOV.txt, + escrow_contacts.daily.gov.GOV.txt, + escrow_domain_statuses.daily.gov.GOV.txt""", + help="""The files used for load_transition_domain migration script. + Must appear IN ORDER and separated by commas: domain_contacts_filename.txt,contacts_filename.txt,domain_statuses_filename.txt - + where... - - domain_contacts_filename is the Data file with domain contact information + - domain_contacts_filename is the Data file with domain contact + information - contacts_filename is the Data file with contact information - - domain_statuses_filename is the Data file with domain status information""", + - domain_statuses_filename is the Data file with domain status + information""", ) parser.add_argument( @@ -196,13 +197,15 @@ class Command(BaseCommand): if len(matching_domain_informations) == 0: TerminalHelper.print_conditional( debug_on, - f"""{TerminalColors.YELLOW}Missing Domain Information{TerminalColors.ENDC}""", + f"""{TerminalColors.YELLOW}Missing Domain Information + {TerminalColors.ENDC}""", ) missing_domain_informations.append(transition_domain_name) if len(matching_domain_invitations) == 0: TerminalHelper.print_conditional( debug_on, - f"""{TerminalColors.YELLOW}Missing Domain Invitation{TerminalColors.ENDC}""", + f"""{TerminalColors.YELLOW}Missing Domain Invitation + {TerminalColors.ENDC}""", ) missing_domain_invites.append(transition_domain_name) @@ -225,22 +228,26 @@ class Command(BaseCommand): logger.info( f"""{TerminalColors.OKGREEN} ============= FINISHED ANALYSIS =============== - + {total_missing_domains} Missing Domains: (These are transition domains that are missing from the Domain Table) - {TerminalColors.YELLOW}{missing_domains_as_string}{TerminalColors.OKGREEN} - + {TerminalColors.YELLOW}{missing_domains_as_string} + {TerminalColors.OKGREEN} {total_duplicate_domains} Duplicate Domains: - (These are transition domains which have duplicate entries in the Domain Table) - {TerminalColors.YELLOW}{duplicate_domains_as_string}{TerminalColors.OKGREEN} - + (These are transition domains which have duplicate + entries in the Domain Table) + {TerminalColors.YELLOW}{duplicate_domains_as_string} + {TerminalColors.OKGREEN} {total_missing_domain_informations} Domain Information Entries missing: - (These are transition domains which have no entries in the Domain Information Table) - {TerminalColors.YELLOW}{missing_domain_informations_as_string}{TerminalColors.OKGREEN} - + (These are transition domains which have no entries + in the Domain Information Table) + {TerminalColors.YELLOW}{missing_domain_informations_as_string} + {TerminalColors.OKGREEN} {total_missing_domain_invitations} Domain Invitations missing: - (These are transition domains which have no entires in the Domain Invitation Table) - {TerminalColors.YELLOW}{missing_domain_invites_as_string}{TerminalColors.OKGREEN} + (These are transition domains which have no entires in + the Domain Invitation Table) + {TerminalColors.YELLOW}{missing_domain_invites_as_string} + {TerminalColors.OKGREEN} {TerminalColors.ENDC} """ ) @@ -377,13 +384,10 @@ class Command(BaseCommand): logger.info( f""" {TerminalColors.YELLOW} - PLEASE Re-Run the script with the correct file location and filenames: - - EXAMPLE: - docker compose run -T app ./manage.py test_domain_migration --runMigrations --migrationDirectory /app/tmp --migrationFilenames escrow_domain_contacts.daily.gov.GOV.txt escrow_contacts.daily.gov.GOV.txt escrow_domain_statuses.daily.gov.GOV.txt - + PLEASE Re-Run the script with the correct + file location and filenames: """ - ) # noqa + ) return # Proceed executing the migration scripts @@ -400,33 +404,6 @@ class Command(BaseCommand): ) self.run_transfer_script(debug_on, prompts_enabled) - def simulate_user_logins(self, debug_on): - """Simulates logins for users (this will add - Domain Information objects to our tables)""" - - # logger.info(f"" - # f"{TerminalColors.OKCYAN}" - # f"================== SIMULATING LOGINS ==================" - # f"{TerminalColors.ENDC}") - - # for invite in DomainInvitation.objects.all(): #TODO: move to unit test - # #DEBUG: - # TerminalHelper.print_conditional(debug_on, - # f"{TerminalColors.OKCYAN}" - # f"Processing invite: {invite}" - # f"{TerminalColors.ENDC}") - # # get a user with this email address - # user, user_created = User.objects.get_or_create(email=invite.email, username=invite.email) - # #DEBUG: - # TerminalHelper.print_conditional(user_created, - # f"""{TerminalColors.OKCYAN}No user found (creating temporary user object){TerminalColors.ENDC}""") - # TerminalHelper.print_conditional(debug_on, - # f"""{TerminalColors.OKCYAN}Executing first-time login for user: {user}{TerminalColors.ENDC}""") - # user.first_login() - # if user_created: - # logger.info(f"""{TerminalColors.YELLOW}(Deleting temporary user object){TerminalColors.ENDC}""") - # user.delete() - def handle( self, **options, @@ -453,9 +430,6 @@ class Command(BaseCommand): debug_on = options.get("debug") prompts_enabled = options.get("prompt") run_migrations_enabled = options.get("runMigrations") - simulate_user_login_enabled = ( - False # TODO: delete? Moving to unit test... options.get("triggerLogins") - ) TerminalHelper.print_conditional( debug_on, @@ -538,30 +512,7 @@ class Command(BaseCommand): ) prompt_continuation_of_analysis = True - # STEP 2 -- SIMULATE LOGINS - # Simulate user login for each user in domain - # invitation if specified by user OR if running - # migration scripts. - # (NOTE: Although users can choose to run login - # simulations separately (for testing purposes), - # if we are running all migration scripts, we should - # automatically execute this as the final step - # to ensure Domain Information objects get added - # to the database.) - - if run_migrations_enabled and simulate_user_login_enabled: - if prompts_enabled: - simulate_user_login_enabled = TerminalHelper.query_yes_no( - f"""{TerminalColors.FAIL} - Proceed with simulating user logins? - {TerminalColors.ENDC}""" - ) - if not simulate_user_login_enabled: - return - self.simulate_user_logins(debug_on) - prompt_continuation_of_analysis = True - - # STEP 3 -- SEND INVITES + # STEP 2 -- SEND INVITES proceed_with_sending_invites = run_migrations_enabled if prompts_enabled and run_migrations_enabled: proceed_with_sending_invites = TerminalHelper.query_yes_no( @@ -574,7 +525,7 @@ class Command(BaseCommand): self.run_send_invites_script(debug_on, prompts_enabled) prompt_continuation_of_analysis = True - # STEP 4 -- ANALYZE TABLES & GENERATE REPORT + # STEP 3 -- ANALYZE TABLES & GENERATE REPORT # Analyze tables for corrupt data... if prompt_continuation_of_analysis and prompts_enabled: # ^ (only prompt if we ran steps 1 and/or 2) diff --git a/src/registrar/management/commands/utility/terminal_helper.py b/src/registrar/management/commands/utility/terminal_helper.py index 1bc2cbb33..7327ef3bd 100644 --- a/src/registrar/management/commands/utility/terminal_helper.py +++ b/src/registrar/management/commands/utility/terminal_helper.py @@ -1,5 +1,4 @@ import logging -import os import sys logger = logging.getLogger(__name__) @@ -23,6 +22,7 @@ class TerminalColors: class TerminalHelper: + @staticmethod def query_yes_no(question: str, default="yes") -> bool: """Ask a yes/no question via raw_input() and return their answer. @@ -53,6 +53,7 @@ class TerminalHelper: else: logger.info("Please respond with 'yes' or 'no' " "(or 'y' or 'n').\n") + @staticmethod def print_conditional(print_condition: bool, print_statement: str): """This function reduces complexity of debug statements in other functions. @@ -62,12 +63,13 @@ class TerminalHelper: if print_condition: logger.info(print_statement) + @staticmethod def prompt_for_execution( system_exit_on_terminate: bool, info_to_inspect: str, prompt_title: str ) -> bool: """Create to reduce code complexity. Prompts the user to inspect the given string - and asks if they wish to execute it. + and asks if they wish to proceed. Returns true if the user responds (y), Returns false if the user responds (n)""" diff --git a/src/registrar/models/domain.py b/src/registrar/models/domain.py index 12cb8b5db..07e49dfdd 100644 --- a/src/registrar/models/domain.py +++ b/src/registrar/models/domain.py @@ -262,8 +262,11 @@ class Domain(TimeStampedModel, DomainHelper): """Creates the host object in the registry doesn't add the created host to the domain returns ErrorCode (int)""" - if addrs is not None: - addresses = [epp.Ip(addr=addr) for addr in addrs] + if addrs is not None and addrs != []: + addresses = [ + epp.Ip(addr=addr, ip="v6" if self.is_ipv6(addr) else None) + for addr in addrs + ] request = commands.CreateHost(name=host, addrs=addresses) else: request = commands.CreateHost(name=host) @@ -274,7 +277,7 @@ class Domain(TimeStampedModel, DomainHelper): return response.code except RegistryError as e: logger.error("Error _create_host, code was %s error was %s" % (e.code, e)) - return e.code + raise e def _convert_list_to_dict(self, listToConvert: list[tuple[str, list]]): """converts a list of hosts into a dictionary @@ -293,14 +296,16 @@ class Domain(TimeStampedModel, DomainHelper): newDict[tup[0]] = tup[1] return newDict - def isSubdomain(self, nameserver: str): + @classmethod + def isSubdomain(cls, name: str, nameserver: str): """Returns boolean if the domain name is found in the argument passed""" subdomain_pattern = r"([\w-]+\.)*" - full_pattern = subdomain_pattern + self.name + full_pattern = subdomain_pattern + name regex = re.compile(full_pattern) return bool(regex.match(nameserver)) - def checkHostIPCombo(self, nameserver: str, ip: list[str]): + @classmethod + def checkHostIPCombo(cls, name: str, nameserver: str, ip: list[str]): """Checks the parameters past for a valid combination raises error if: - nameserver is a subdomain but is missing ip @@ -314,22 +319,23 @@ class Domain(TimeStampedModel, DomainHelper): NameserverError (if exception hit) Returns: None""" - if self.isSubdomain(nameserver) and (ip is None or ip == []): + if cls.isSubdomain(name, nameserver) and (ip is None or ip == []): raise NameserverError(code=nsErrorCodes.MISSING_IP, nameserver=nameserver) - elif not self.isSubdomain(nameserver) and (ip is not None and ip != []): + elif not cls.isSubdomain(name, nameserver) and (ip is not None and ip != []): raise NameserverError( code=nsErrorCodes.GLUE_RECORD_NOT_ALLOWED, nameserver=nameserver, ip=ip ) elif ip is not None and ip != []: for addr in ip: - if not self._valid_ip_addr(addr): + if not cls._valid_ip_addr(addr): raise NameserverError( code=nsErrorCodes.INVALID_IP, nameserver=nameserver, ip=ip ) return None - def _valid_ip_addr(self, ipToTest: str): + @classmethod + def _valid_ip_addr(cls, ipToTest: str): """returns boolean if valid ip address string We currently only accept v4 or v6 ips returns: @@ -382,7 +388,9 @@ class Domain(TimeStampedModel, DomainHelper): if newHostDict[prevHost] is not None and set( newHostDict[prevHost] ) != set(addrs): - self.checkHostIPCombo(nameserver=prevHost, ip=newHostDict[prevHost]) + self.__class__.checkHostIPCombo( + name=self.name, nameserver=prevHost, ip=newHostDict[prevHost] + ) updated_values.append((prevHost, newHostDict[prevHost])) new_values = { @@ -392,7 +400,9 @@ class Domain(TimeStampedModel, DomainHelper): } for nameserver, ip in new_values.items(): - self.checkHostIPCombo(nameserver=nameserver, ip=ip) + self.__class__.checkHostIPCombo( + name=self.name, nameserver=nameserver, ip=ip + ) return (deleted_values, updated_values, new_values, previousHostDict) @@ -567,7 +577,11 @@ class Domain(TimeStampedModel, DomainHelper): if len(hosts) > 13: raise NameserverError(code=nsErrorCodes.TOO_MANY_HOSTS) - if self.state not in [self.State.DNS_NEEDED, self.State.READY]: + if self.state not in [ + self.State.DNS_NEEDED, + self.State.READY, + self.State.UNKNOWN, + ]: raise ActionNotAllowed("Nameservers can not be " "set in the current state") logger.info("Setting nameservers") @@ -1351,7 +1365,7 @@ class Domain(TimeStampedModel, DomainHelper): @transition( field="state", - source=[State.DNS_NEEDED], + source=[State.DNS_NEEDED, State.READY], target=State.READY, # conditions=[dns_not_needed] ) @@ -1514,7 +1528,7 @@ class Domain(TimeStampedModel, DomainHelper): data = registry.send(req, cleaned=True).res_data[0] host = { "name": name, - "addrs": getattr(data, "addrs", ...), + "addrs": [item.addr for item in getattr(data, "addrs", [])], "cr_date": getattr(data, "cr_date", ...), "statuses": getattr(data, "statuses", ...), "tr_date": getattr(data, "tr_date", ...), @@ -1539,10 +1553,9 @@ class Domain(TimeStampedModel, DomainHelper): return [] for ip_addr in ip_list: - if self.is_ipv6(ip_addr): - edited_ip_list.append(epp.Ip(addr=ip_addr, ip="v6")) - else: # default ip addr is v4 - edited_ip_list.append(epp.Ip(addr=ip_addr)) + edited_ip_list.append( + epp.Ip(addr=ip_addr, ip="v6" if self.is_ipv6(ip_addr) else None) + ) return edited_ip_list @@ -1580,7 +1593,7 @@ class Domain(TimeStampedModel, DomainHelper): return response.code except RegistryError as e: logger.error("Error _update_host, code was %s error was %s" % (e.code, e)) - return e.code + raise e def addAndRemoveHostsFromDomain( self, hostsToAdd: list[str], hostsToDelete: list[str] diff --git a/src/registrar/templates/domain_dnssec.html b/src/registrar/templates/domain_dnssec.html index 691ba79b2..c92ca2b78 100644 --- a/src/registrar/templates/domain_dnssec.html +++ b/src/registrar/templates/domain_dnssec.html @@ -7,7 +7,7 @@
DNSSEC, or DNS Security Extensions, is additional security layer to protect your website. Enabling DNSSEC ensures that when someone visits your website, they can be certain that it’s connecting to the correct server, preventing potential hijacking or tampering with your domain's records.
+DNSSEC, or DNS Security Extensions, is an additional security layer to protect your website. Enabling DNSSEC ensures that when someone visits your domain, they can be certain that it’s connecting to the correct server, preventing potential hijacking or tampering with your domain's records.