zydronium/manage.get.gov
1
0
Fork 0
mirror of https://github.com/cisagov/manage.get.gov.git synced 2025-05-31 01:33:56 +02:00
Code Issues Projects Releases Packages Wiki Activity

added readme

Browse source 
Signed-off-by: CocoByte <nicolle.leclair@gmail.com>
This commit is contained in:
CocoByte 2023-10-12 00:06:59 -06:00
parent d02f1acf34
commit 9ab908119d
No known key found for this signature in database
GPG key ID: BBFAA2526384C97F
2 changed files with 136 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

52
docs/operations/data_migration.md
View file

@ -76,3 +76,55 @@ An example script using this technique is in
```shell ```shell
docker compose run app ./manage.py load_domain_invitations /app/escrow_domain_contacts.daily.dotgov.GOV.txt /app/escrow_contacts.daily.dotgov.GOV.txt docker compose run app ./manage.py load_domain_invitations /app/escrow_domain_contacts.daily.dotgov.GOV.txt /app/escrow_contacts.daily.dotgov.GOV.txt
``` ```
## Transition Domains
Verisign provides information about Transition Domains in 3 files;
FILE 1: escrow_domain_contacts.daily.gov.GOV.txt
FILE 2: escrow_contacts.daily.gov.GOV.txt
FILE 3: escrow_domain_statuses.daily.gov.GOV.txt
Transferring this data from these files into our domain tables happens in two steps;
### STEP 1: Load Transition Domain data into TransitionDomain table
**SETUP**
In order to use the management command, we need to add the files to a folder under `src/`.
This will allow Docker to mount the files to a container (under `/app`) for our use.
- Create a folder called `tmp` underneath `src/`
- Add the above files to this folder
- Open a terminal and navigate to `src/`
Then run the following command (This will parse the three files in your `tmp` folder and load the information into the TransitionDomain table);
```shell
docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt
```
**OPTIONAL COMMAND LINE ARGUMENTS**:
`--debug`
This will print out additional, detailed logs.
`--limitParse 100`
Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes.
`--resetTable`
This will delete all the data loaded into transtion_domain. It is helpful if you want to see the entries reload from scratch or for clearing test data.
### STEP 2: Transfer Transition Domain data into main Domain tables
Now that we've loaded all the data into TransitionDomain, we need to update the main Domain and DomainInvitation tables with this information.
In the same terminal as used in STEP 1, run the command below;
(This will parse the data in TransitionDomain and either create a corresponding Domain object, OR, if a corresponding Domain already exists, it will update that Domain with the incoming status. It will also create DomainInvitation objects for each user associated with the domain):
```shell
docker compose run -T app ./manage.py transfer_transition_domains_to_domains
```
**OPTIONAL COMMAND LINE ARGUMENTS**:
`--debug`
This will print out additional, detailed logs.
`--limitParse 100`
Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes.

138
src/registrar/management/commands/load_transition_domain.py
View file

@ -267,8 +267,40 @@ class Command(BaseCommand):
{domains_without_status_as_string} {domains_without_status_as_string}
{termColors.ENDC}""" {termColors.ENDC}"""
) )
def handle( # noqa: C901 def print_debug(self, print_condition: bool, print_statement: str):
"""This function reduces complexity of debug statements
in other functions.
It uses the logger to write the given print_statement to the
terminal if print_condition is TRUE"""
# DEBUG:
if print_condition:
logger.info(print_statement)
def prompt_table_reset():
"""Brings up a prompt in the terminal asking
if the user wishes to delete data in the
TransitionDomain table. If the user confirms,
deletes all the data in the TransitionDomain table"""
confirm_reset = query_yes_no(
f"""
{termColors.FAIL}
WARNING: Resetting the table will permanently delete all
the data!
Are you sure you want to continue?{termColors.ENDC}"""
)
if confirm_reset:
logger.info(
f"""{termColors.YELLOW}
----------Clearing Table Data----------
(please wait)
{termColors.ENDC}"""
)
TransitionDomain.objects.all().delete()
def handle(
self, self,
domain_contacts_filename, domain_contacts_filename,
contacts_filename, contacts_filename,
@ -281,21 +313,7 @@ class Command(BaseCommand):
# If --resetTable was used, prompt user to confirm # If --resetTable was used, prompt user to confirm
# deletion of table data # deletion of table data
if options.get("resetTable"): if options.get("resetTable"):
confirm_reset = query_yes_no( self.prompt_table_reset()
f"""
{termColors.FAIL}
WARNING: Resetting the table will permanently delete all
the data!
Are you sure you want to continue?{termColors.ENDC}"""
)
if confirm_reset:
logger.info(
f"""{termColors.YELLOW}
----------Clearing Table Data----------
(please wait)
{termColors.ENDC}"""
)
TransitionDomain.objects.all().delete()
# Get --debug argument # Get --debug argument
debug_on = options.get("debug") debug_on = options.get("debug")
@ -354,26 +372,35 @@ class Command(BaseCommand):
# fields are just domain, userid, role # fields are just domain, userid, role
# lowercase the domain names # lowercase the domain names
new_entry_domainName = row[0].lower() new_entry_domain_name = row[0].lower()
user_id = row[1] user_id = row[1]
new_entry_status = TransitionDomain.StatusChoices.READY new_entry_status = TransitionDomain.StatusChoices.READY
new_entry_email = "" new_entry_email = ""
new_entry_emailSent = False # set to False by default new_entry_emailSent = False # set to False by default
if new_entry_domainName not in domain_status_dictionary: # PART 1: Get the status
# this domain has no status...default to "Create" if new_entry_domain_name not in domain_status_dictionary:
if new_entry_domainName not in domains_without_status: # This domain has no status...default to "Create"
domains_without_status.append(new_entry_domainName) # (For data analysis purposes, add domain name
# to list of all domains without status
# (avoid duplicate entries))
if new_entry_domain_name not in domains_without_status:
domains_without_status.append(new_entry_domain_name)
else: else:
original_status = domain_status_dictionary[new_entry_domainName] # Map the status
original_status = domain_status_dictionary[new_entry_domain_name]
mapped_status = self.get_mapped_status(original_status) mapped_status = self.get_mapped_status(original_status)
if mapped_status is None: if mapped_status is None:
# (For data analysis purposes, check for any statuses
# that don't have a mapping and add to list
# of "outlier statuses")
logger.info("Unknown status: " + original_status) logger.info("Unknown status: " + original_status)
outlier_statuses.append(original_status) outlier_statuses.append(original_status)
else: else:
new_entry_status = mapped_status new_entry_status = mapped_status
# PART 2: Get the e-mail
if user_id not in user_emails_dictionary: if user_id not in user_emails_dictionary:
# this user has no e-mail...this should never happen # this user has no e-mail...this should never happen
if user_id not in users_without_email: if user_id not in users_without_email:
@ -381,6 +408,7 @@ class Command(BaseCommand):
else: else:
new_entry_email = user_emails_dictionary[user_id] new_entry_email = user_emails_dictionary[user_id]
# PART 3: Create the transition domain object
# Check for duplicate data in the file we are # Check for duplicate data in the file we are
# parsing so we do not add duplicates # parsing so we do not add duplicates
# NOTE: Currently, we allow duplicate domains, # NOTE: Currently, we allow duplicate domains,
@ -389,7 +417,7 @@ class Command(BaseCommand):
# since we are still deciding on whether # since we are still deciding on whether
# to make this field unique or not. ~10/25/2023 # to make this field unique or not. ~10/25/2023
existing_domain = next( existing_domain = next(
(x for x in to_create if x.domain_name == new_entry_domainName), (x for x in to_create if x.domain_name == new_entry_domain_name),
None, None,
) )
existing_domain_user_pair = next( existing_domain_user_pair = next(
@ -397,43 +425,48 @@ class Command(BaseCommand):
x x
for x in to_create for x in to_create
if x.username == new_entry_email if x.username == new_entry_email
and x.domain_name == new_entry_domainName and x.domain_name == new_entry_domain_name
), ),
None, None,
) )
if existing_domain is not None: if existing_domain is not None:
if debug_on: # DEBUG:
logger.info( self.print_debug(
f"{termColors.YELLOW} DUPLICATE Verisign entries found for domain: {new_entry_domainName} {termColors.ENDC}" # noqa debug_on,
f"{termColors.YELLOW} DUPLICATE Verisign entries found for domain: {new_entry_domain_name} {termColors.ENDC}" # noqa
) )
if new_entry_domainName not in duplicate_domains: if new_entry_domain_name not in duplicate_domains:
duplicate_domains.append(new_entry_domainName) duplicate_domains.append(new_entry_domain_name)
if existing_domain_user_pair is not None: if existing_domain_user_pair is not None:
if debug_on: # DEBUG:
logger.info( self.print_debug(
f"""{termColors.YELLOW} DUPLICATE Verisign entries found for domain - user {termColors.BackgroundLightYellow} PAIR {termColors.ENDC}{termColors.YELLOW}: debug_on,
{new_entry_domainName} - {new_entry_email} {termColors.ENDC}""" # noqa f"""{termColors.YELLOW} DUPLICATE Verisign entries found for domain - user {termColors.BackgroundLightYellow} PAIR {termColors.ENDC}{termColors.YELLOW}:
{new_entry_domain_name} - {new_entry_email} {termColors.ENDC}""" # noqa
) )
if existing_domain_user_pair not in duplicate_domain_user_combos: if existing_domain_user_pair not in duplicate_domain_user_combos:
duplicate_domain_user_combos.append(existing_domain_user_pair) duplicate_domain_user_combos.append(existing_domain_user_pair)
else: else:
try: try:
existing_entry = TransitionDomain.objects.get( entry_exists = TransitionDomain.objects.exists(
username=new_entry_email, domain_name=new_entry_domainName username=new_entry_email, domain_name=new_entry_domain_name
) )
if(entry_exists):
existing_entry = TransitionDomain.objects.get(
username=new_entry_email, domain_name=new_entry_domain_name
)
if existing_entry.status != new_entry_status: if existing_entry.status != new_entry_status:
# DEBUG: # DEBUG:
if debug_on: self.print_debug(
logger.info( debug_on,
f"{termColors.OKCYAN}" f"{termColors.OKCYAN}"
f"Updating entry: {existing_entry}" f"Updating entry: {existing_entry}"
f"Status: {existing_entry.status} > {new_entry_status}" # noqa f"Status: {existing_entry.status} > {new_entry_status}" # noqa
f"Email Sent: {existing_entry.email_sent} > {new_entry_emailSent}" # noqa f"Email Sent: {existing_entry.email_sent} > {new_entry_emailSent}" # noqa
f"{termColors.ENDC}" f"{termColors.ENDC}"
) )
existing_entry.status = new_entry_status
existing_entry.status = new_entry_status
existing_entry.email_sent = new_entry_emailSent existing_entry.email_sent = new_entry_emailSent
existing_entry.save() existing_entry.save()
@ -441,7 +474,7 @@ class Command(BaseCommand):
# no matching entry, make one # no matching entry, make one
new_entry = TransitionDomain( new_entry = TransitionDomain(
username=new_entry_email, username=new_entry_email,
domain_name=new_entry_domainName, domain_name=new_entry_domain_name,
status=new_entry_status, status=new_entry_status,
email_sent=new_entry_emailSent, email_sent=new_entry_emailSent,
) )
@ -449,26 +482,23 @@ class Command(BaseCommand):
total_new_entries += 1 total_new_entries += 1
# DEBUG: # DEBUG:
if debug_on: self.print_debug(
logger.info( debug_on,
f"{termColors.OKCYAN} Adding entry {total_new_entries}: {new_entry} {termColors.ENDC}" # noqa f"{termColors.OKCYAN} Adding entry {total_new_entries}: {new_entry} {termColors.ENDC}" # noqa
) )
except TransitionDomain.MultipleObjectsReturned: except TransitionDomain.MultipleObjectsReturned:
logger.info( logger.info(
f"{termColors.FAIL}" f"{termColors.FAIL}"
f"!!! ERROR: duplicate entries exist in the" f"!!! ERROR: duplicate entries exist in the"
f"transtion_domain table for domain:" f"transtion_domain table for domain:"
f"{new_entry_domainName}" f"{new_entry_domain_name}"
f"----------TERMINATING----------" f"----------TERMINATING----------"
) )
sys.exit() sys.exit()
# DEBUG: # DEBUG:
if debug_on or debug_max_entries_to_parse > 0: if (total_rows_parsed >= debug_max_entries_to_parse
if ( and debug_max_entries_to_parse != 0):
total_rows_parsed >= debug_max_entries_to_parse
and debug_max_entries_to_parse != 0
):
logger.info( logger.info(
f"{termColors.YELLOW}" f"{termColors.YELLOW}"
f"----PARSE LIMIT REACHED. HALTING PARSER.----" f"----PARSE LIMIT REACHED. HALTING PARSER.----"