zydronium/manage.get.gov
1
0
Fork 0
mirror of https://github.com/cisagov/manage.get.gov.git synced 2025-05-31 01:33:56 +02:00
Code Issues Projects Releases Packages Wiki Activity

Changed "--prompt" to "--disablePrompts". Updates to documentation. Needs testing

Browse source
This commit is contained in:
CocoByte 2023-11-08 14:37:17 -06:00
parent 001da13207
commit ea5df7adee
No known key found for this signature in database
GPG key ID: BBFAA2526384C97F
4 changed files with 123 additions and 96 deletions
Download patch file Download diff file Expand all files Collapse all files

121
docs/operations/data_migration.md
View file

@ -79,11 +79,36 @@ docker compose run app ./manage.py load_domain_invitations /app/escrow_domain_co
```
## Transition Domains (Part 1) - Setup Files for Import
We are provided with information about Transition Domains in 3 files:
#### STEP 1: obtain data files
We are provided with information about Transition Domains in the following files:
- FILE 1: **escrow_domain_contacts.daily.gov.GOV.txt** -> has the map of domain names to contact ID. Domains in this file will usually have 3 contacts each
- FILE 2: **escrow_contacts.daily.gov.GOV.txt** -> has the mapping of contact id to contact email address (which is what we care about for sending domain invitations)
- FILE 3: **escrow_domain_statuses.daily.gov.GOV.txt** -> has the map of domains and their statuses
- FILE 4: **escrow_domains.daily.dotgov.GOV.txt** -> has basic domain data
- FILE 5: **domainadditionaldatalink.adhoc.dotgov.txt** -> has the map of domains to other data like authority, organization, & domain type
- FILE 6: **domaintypes.adhoc.dotgov.txt** -> has domain type data
- FILE 7: **organization.adhoc.dotgov.txt** -> has organization data
- FILE 8: **authority.adhoc.dotgov.txt** -> has authority data
- FILE 9: **agency.adhoc.dotgov.txt** -> has agency data
#### STEP 2: obtain JSON file (for file locations)
Add a JSON file called "migrationFilepaths.json" with the following contents (update filenames and directory as needed):
```
{
"directory": "migrationdata",
"contacts_filename": "escrow_contacts.daily.dotgov.GOV.txt",
"domain_contacts_filename": "escrow_domain_contacts.daily.dotgov.GOV.txt",
"domain_statuses_filename": "escrow_domain_statuses.daily.dotgov.GOV.txt",
"domain_escrow_filename": "escrow_domains.daily.dotgov.GOV.txt",
"domain_additional_filename": "domainadditionaldatalink.adhoc.dotgov.txt",
"domain_adhoc_filename": "domaintypes.adhoc.dotgov.txt",
"organization_adhoc_filename": "organization.adhoc.dotgov.txt"
"authority_adhoc_filename": "authority.adhoc.dotgov.txt",
"agency_adhoc_filename": "agency.adhoc.dotgov.txt",
}
```
We need to run a few scripts to parse these files into our domain tables.
We can do this both locally and in a sandbox.
@ -203,7 +228,7 @@ cat ../tmp/{filename} > migrationdata/{filename}
In order to run the scripts locally, we need to add the files to a folder under `src/`.
This will allow Docker to mount the files to a container (under `/app`) for our use.
- Add the above files to the `migrationdata/` folder
- Add the same files from section 1 to a TEMPORARY `tmp/` folder under `src/` (do not check this folder into our repo)
- Open a terminal and navigate to `src/`
@ -216,9 +241,13 @@ This will allow Docker to mount the files to a container (under `/app`) for our
Run the following command, making sure the file paths point to the right location. This will parse the three given files and load the information into the TransitionDomain table.
(NOTE: If working in cloud.gov, change "/app/tmp" to point to the `migrationdata/` directory and and remove "docker compose run -T app" from the command)
##### LOCAL COMMAND
```shell
docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt --debug
docker-compose exec app ./manage.py load_transition_domain migrationFilepaths.json --directory /app/tmp --debug
```
##### SANDBOX COMMAND
```shell
./manage.py load_transition_domain migrationFilepaths.json --debug
```
##### COMMAND LINE ARGUMENTS:
@ -232,6 +261,41 @@ Directs the script to load only the first 100 entries into the table. You can a
`--resetTable`
This will delete all the data in transtion_domain. It is helpful if you want to see the entries reload from scratch or for clearing test data.
###### (arguments that override filepaths and directories if needed)
`--infer_filenames`
Determines if we should infer filenames or not. Recommended to be enabled only in a development or testing setting..
`--directory`
Defines the directory where all data files and the JSON are stored.
`--domain_contacts_filename`
Defines the filename for domain contact information.
`--contacts_filename`
Defines the filename for contact information.
`--domain_statuses_filename`
Defines the filename for domain status information.
`--agency_adhoc_filename`
Defines the filename for agency adhocs.
`--domain_additional_filename`
Defines the filename for additional domain data.
`--domain_escrow_filename`
Defines the filename for creation/expiration domain data.
`--domain_adhoc_filename`
Defines the filename for domain type adhocs.
`--organization_adhoc_filename`
Defines the filename for domain type adhocs.
`--authority_adhoc_filename`
Defines the filename for domain type adhocs.
### STEP 2: Transfer Transition Domain data into main Domain tables
@ -239,10 +303,14 @@ Now that we've loaded all the data into TransitionDomain, we need to update the
In the same terminal as used in STEP 1, run the command below;
(This will parse the data in TransitionDomain and either create a corresponding Domain object, OR, if a corresponding Domain already exists, it will update that Domain with the incoming status. It will also create DomainInvitation objects for each user associated with the domain):
(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
##### LOCAL COMMAND
```shell
docker compose run -T app ./manage.py transfer_transition_domains_to_domains --debug
```
##### SANDBOX COMMAND
```shell
./manage.py transfer_transition_domains_to_domains --debug
```
##### COMMAND LINE ARGUMENTS:
@ -256,10 +324,14 @@ Directs the script to load only the first 100 entries into the table. You can a
To send invitation emails for every transition domain in the transition domain table, execute the following command:
(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
##### LOCAL COMMAND
```shell
docker compose run -T app ./manage.py send_domain_invitations -s
```
##### SANDBOX COMMAND
```shell
./manage.py send_domain_invitations -s
```
### STEP 4: Test the results (Run the analyzer script)
@ -269,18 +341,27 @@ This script's main function is to scan the transition domain and domain tables f
To analyze our database without running migrations, execute the script without any optional arguments:
(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
##### LOCAL COMMAND
```shell
docker compose run -T app ./manage.py master_domain_migrations --debug
```
##### SANDBOX COMMAND
```shell
./manage.py master_domain_migrations --debug
```
#### OPTION 2 - RUN MIGRATIONS FEATURE
To run the migrations again (all above migration steps) before analyzing, execute the following command (read the documentation on the terminal arguments below. Everything used by the migration scripts can also be passed into this script and will have the same effects). NOTE: --debug and --prompt allow you to step through the migration process and exit it after each step if you need to. It is recommended that you use these arguments when using the --runMigrations feature:
To run the migrations again (all above migration steps) before analyzing, execute the following command (read the documentation on the terminal arguments below. Everything used by the migration scripts can also be passed into this script and will have the same effects). NOTE: --debug provides detailed logging statements during the migration. It is recommended that you use this argument when using the --runMigrations feature:
(NOTE: If working in cloud.gov, and remove "docker compose run -T app" from the command)
(NOTE: If you named your JSON file something other than "migrationFilepaths.json" (all the way back in the "file setup" section). You will want to utilize the `--migrationJSON` argument in the following commands...)
##### LOCAL COMMAND
```shell
docker compose run -T app ./manage.py master_domain_migrations --runMigrations --debug --prompt
docker compose run -T app ./manage.py master_domain_migrations --migrationDirectory /app/tmp --runMigrations --debug
```
##### SANDBOX COMMAND
```shell
./manage.py master_domain_migrations --runMigrations --debug
```
##### COMMAND LINE ARGUMENTS
@ -291,25 +372,18 @@ Runs all scripts (in sequence) for transition domain migrations
`--migrationDirectory`
The location of the files used for load_transition_domain migration script.
The location of both the JSON file and all files needed for migration.
(default is "migrationdata" (This is the sandbox directory))
Example Usage:
*--migrationDirectory /app/tmp*
`--migrationFilenames`
`--migrationJSON`
The filenames used for load_transition_domain migration script.
Must appear *in oprder* and comma-delimited:
default is "escrow_domain_contacts.daily.gov.GOV.txt,escrow_contacts.daily.gov.GOV.txt,escrow_domain_statuses.daily.gov.GOV.txt"
where...
- domain_contacts_filename is the Data file with domain contact information
- contacts_filename is the Data file with contact information
- domain_statuses_filename is the Data file with domain status information
The filename of the JSON that holds all the filepath info needed for migrations.
Example Usage:
*--migrationFilenames domain_contacts_filename.txt,contacts_filename.txt,domain_statuses_filename.txt*
*--migrationJSON migrationFilepaths.json*
`--sep`
@ -320,9 +394,10 @@ Delimiter for the migration scripts to correctly parse the given text files.
Activates additional print statements
`--prompt`
`--disablePrompts`
Activates terminal prompts that allows the user to step through each portion of this script.
Disables the terminal prompts that allows the user to step through each portion of this script.
*used to facilitate unit tests. Not recommended for everyday use*
`--limitParse`

3
src/registrar/management/commands/load_transition_domain.py
View file

@ -63,9 +63,6 @@ class Command(BaseCommand):
action=argparse.BooleanOptionalAction,
)
# TODO - Narrow this down
# TODO - this isn't pulling in the directory from the master script. Needs to be corrected @Nicolle - todo
# default="/app/tmp"
parser.add_argument(
"--directory", default="migrationdata", help="Desired directory"
)

94
src/registrar/management/commands/master_domain_migrations.py
View file

@ -45,27 +45,21 @@ class Command(BaseCommand):
The location of the files used for load_transition_domain migration script
EXAMPLE USAGE:
> --migrationDirectory /app/tmp
--migrationFilenames
The files used for load_transition_domain migration script.
Must appear IN ORDER and comma-delimiteds:
--migrationJSON
The name of the JSON file used for load_transition_domain migration script
EXAMPLE USAGE:
> --migrationFilenames domain_contacts_filename.txt,contacts_filename.txt,domain_statuses_filename.txt
where...
- domain_contacts_filename is the Data file with domain contact information
- contacts_filename is the Data file with contact information
- domain_statuses_filename is the Data file with domain status information
> --migrationJSON migrationFilepaths.json
--sep
Delimiter for the migration scripts to correctly parse the given text files.
(usually this can remain at default value of |)
--debug
A boolean (default to true), which activates additional print statements
Activates additional print statements
--prompt
A boolean (default to true), which activates terminal prompts
that allows the user to step through each portion of this
--disablePrompts
Disables terminal prompts that allows the user to step through each portion of this
script.
--limitParse
@ -99,7 +93,8 @@ class Command(BaseCommand):
# TODO: make this a mandatory argument (if/when we strip out defaults, it will be mandatory)
# TODO: use the migration directory arg or force user to type FULL filepath?
parser.add_argument(
"migration_json_filename",
"--migrationJSON",
default="migrationFilepaths.json",
help=(
"A JSON file that holds the location and filenames"
"of all the data files used for migrations"
@ -116,31 +111,13 @@ class Command(BaseCommand):
),
)
# TODO: deprecate this once JSON module is done? (or keep as an override)
parser.add_argument(
"--migrationFilenames",
default="escrow_domain_contacts.daily.gov.GOV.txt,"
"escrow_contacts.daily.gov.GOV.txt,"
"escrow_domain_statuses.daily.gov.GOV.txt",
help="""The files used for load_transition_domain migration script.
Must appear IN ORDER and separated by commas:
domain_contacts_filename.txt,contacts_filename.txt,domain_statuses_filename.txt
where...
- domain_contacts_filename is the Data file with domain contact
information
- contacts_filename is the Data file with contact information
- domain_statuses_filename is the Data file with domain status
information""",
)
parser.add_argument(
"--sep", default="|", help="Delimiter character for the migration files"
)
parser.add_argument("--debug", action=argparse.BooleanOptionalAction)
parser.add_argument("--prompt", action=argparse.BooleanOptionalAction)
parser.add_argument("--disablePrompts", action=argparse.BooleanOptionalAction)
parser.add_argument(
"--limitParse", default=0, help="Sets max number of entries to load"
@ -282,19 +259,23 @@ class Command(BaseCommand):
def run_load_transition_domain_script(
self,
migration_json_filename: str,
file_location: str,
file_directory: str,
sep: str,
reset_table: bool,
debug_on: bool,
prompts_enabled: bool,
debug_max_entries_to_parse: int,
):
if file_directory and file_directory[-1] != "/":
file_directory += "/"
json_filepath = file_directory+migration_json_filename
"""Runs the load_transition_domain script"""
# Create the command string
command_script = "load_transition_domain"
command_string = (
f"./manage.py {command_script} "
f"{file_location+migration_json_filename} "
f"{json_filepath} "
)
if sep is not None and sep != "|":
command_string += f"--sep {sep} "
@ -318,12 +299,12 @@ class Command(BaseCommand):
if proceed or not prompts_enabled:
call_command(
command_script,
f"{file_location+migration_json_filename}",
json_filepath,
sep=sep,
resetTable=reset_table,
debug=debug_on,
limitParse=debug_max_entries_to_parse,
directory=file_location
directory=file_directory
)
def run_transfer_script(self, debug_on: bool, prompts_enabled: bool):
@ -367,9 +348,6 @@ class Command(BaseCommand):
self,
migration_json_filename,
file_location,
domain_contacts_filename,
contacts_filename,
domain_statuses_filename,
sep,
reset_table,
debug_on,
@ -392,10 +370,8 @@ class Command(BaseCommand):
The migration scripts are looking in directory....
{file_location}
....for the following files:
- domain contacts: {domain_contacts_filename}
- contacts: {contacts_filename}
- domain statuses: {domain_statuses_filename}
....for the following JSON:
{migration_json_filename}
{TerminalColors.FAIL}
Does this look correct?{TerminalColors.ENDC}"""
@ -410,7 +386,7 @@ class Command(BaseCommand):
f"""
{TerminalColors.YELLOW}
PLEASE Re-Run the script with the correct
file location and filenames:
JSON filename and directory:
"""
)
return
@ -429,7 +405,6 @@ class Command(BaseCommand):
def handle(
self,
migration_json_filename,
**options,
):
"""
@ -452,7 +427,7 @@ class Command(BaseCommand):
# Get arguments
debug_on = options.get("debug")
prompts_enabled = options.get("prompt")
prompts_enabled = not options.get("disablePrompts")
run_migrations_enabled = options.get("runMigrations")
TerminalHelper.print_conditional(
@ -501,34 +476,13 @@ class Command(BaseCommand):
debug_max_entries_to_parse = int(options.get("limitParse"))
# Grab filepath information from the arguments
file_location = options.get("migrationDirectory") + "/"
filenames = options.get("migrationFilenames").split(",")
if len(filenames) < 3:
filenames_as_string = "{}".format(", ".join(map(str, filenames)))
logger.info(
f"""
{TerminalColors.FAIL}
--migrationFilenames expected 3 filenames to follow it,
but only {len(filenames)} were given:
{filenames_as_string}
PLEASE MODIFY THE SCRIPT AND TRY RUNNING IT AGAIN
============= TERMINATING =============
{TerminalColors.ENDC}
"""
)
sys.exit()
domain_contacts_filename = filenames[0]
contacts_filename = filenames[1]
domain_statuses_filename = filenames[2]
file_location = options.get("migrationDirectory")
migration_json_filename = options.get("migrationJSON")
# Run migration scripts
self.run_migration_scripts(
migration_json_filename,
file_location,
domain_contacts_filename,
contacts_filename,
domain_statuses_filename,
sep,
reset_table,
debug_on,

1
src/registrar/tests/test_transition_domain_migrations.py
View file

@ -63,6 +63,7 @@ class TestMigrations(TestCase):
runMigrations=True,
migrationDirectory=self.test_data_file_location,
migration_json_filename=self.migration_json_filename,
disablePrompts=True
)
def compare_tables(