zydronium/manage.get.gov
1
0
Fork 0
mirror of https://github.com/cisagov/manage.get.gov.git synced 2025-08-04 17:01:56 +02:00
Code Issues Projects Releases Packages Wiki Activity

Updated documentation

Browse source
This commit is contained in:
CocoByte 2023-10-31 12:08:08 -06:00
parent e1257608db
commit 81cc16d1b3
No known key found for this signature in database
GPG key ID: BBFAA2526384C97F
1 changed files with 90 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

113
docs/operations/data_migration.md
View file

@ -87,12 +87,12 @@ FILE 3: **escrow_domain_statuses.daily.gov.GOV.txt** -> has the map of domains a
We need to run a few scripts to parse these files into our domain tables.
We can do this both locally and in a sandbox.
## OPTION 1: SANDBOX
## Load migration data onto a production or sandbox environment
### SANDBOX MIGRATION SETUP
### Load migration data onto a production or sandbox environment
**WARNING:** All files uploaded in this manner are temporary, i.e. they will be deleted when the app is restaged.
Do not use this method to store data you want to keep around permanently.
### STEP 1: Use scp to transfer data
#### STEP 1: Use scp to transfer data
CloudFoundry supports scp as means of transferring data locally to our environment. If you are dealing with a batch of files, try sending across a tar.gz and unpacking that.
**Login to Cloud.gov**
@ -134,7 +134,7 @@ Copy this code into the password prompt from earlier.
NOTE: You can use different utilities to copy this onto the clipboard for you. If you are on Windows, try the command `cf ssh-code | clip`. On Mac, this will be `cf ssh-code | pbcopy`
### STEP 2: Transfer uploaded files to the getgov directory
#### STEP 2: Transfer uploaded files to the getgov directory
Due to the nature of how Cloud.gov operates, the getgov directory is dynamically generated whenever the app is built under the tmp/ folder. We can directly upload files to the tmp/ folder but cannot target the generated getgov folder directly, as we need to spin up a shell to access this. From here, we can move those uploaded files into the getgov directory using the `cat` command. Note that you will have to repeat this for each file you want to move, so it is better to use a tar.gz for multiple, and unpack it inside of the `datamigration` folder.
**SSH into your sandbox**
@ -168,7 +168,7 @@ tar -xvf migrationdata/{FILE_NAME}.tar.gz -C migrationdata/ --strip-components=1
#### Manual method
##### Manual method
If the `cat_files_into_getgov.py` script isn't working, follow these steps instead.
**Move the desired file into the correct directory**
@ -178,34 +178,37 @@ cat ../tmp/{filename} > migrationdata/{filename}
```
### STEP 3: Load Transition Domain data into TransitionDomain table
Run the following script to transfer the existing data on our .txt files to our DB.
```shell
./manage.py load_transition_domain migrationdata/escrow_domain_contacts.daily.gov.GOV.txt migrationdata/escrow_contacts.daily.gov.GOV.txt migrationdata/escrow_domain_statuses.daily.gov.GOV.txt
#### You are now ready to run migration scripts (see "Running the Migration Scripts")
```
## OPTION 2: LOCAL
## Load migration data onto our local environments
```
### LOCAL MIGRATION SETUP (TESTING PURPOSES ONLY)
### Load migration data onto our local environments
Transferring this data from these files into our domain tables happens in two steps;
***IMPORTANT: only use test data, to avoid publicizing PII in our public repo.***
***IMPORTANT: only run the following locally, to avoid publicizing PII in our public repo.***
### STEP 1: Load Transition Domain data into TransitionDomain table
**SETUP**
In order to use the management command, we need to add the files to a folder under `src/`.
In order to run the scripts locally, we need to add the files to a folder under `src/`.
This will allow Docker to mount the files to a container (under `/app`) for our use.
- Create a folder called `tmp` underneath `src/`
- Add the above files to this folder
- Open a terminal and navigate to `src/`
Then run the following command (This will parse the three files in your `tmp` folder and load the information into the TransitionDomain table);
#### You are now ready to run migration scripts (see "Running the Migration Scripts")
### RUNNING THE MIGRATION SCRIPTS
**NOTE: Whil we recommend executing the following scripts individually (Steps 1-3), migrations can also be done 'all at once' using the "Run Migration Feature" in step 4. Use with discretion.**
#### 1 - Load Transition Domains
Run the following command (This will parse the three files in your `tmp` folder and load the information into the TransitionDomain table);
```shell
docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt
docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt --debug
```
**For Sandbox**:
Change "/app/tmp" to point to the sandbox directory
**OPTIONAL COMMAND LINE ARGUMENTS**:
`--debug`
This will print out additional, detailed logs.
@ -214,7 +217,7 @@ This will print out additional, detailed logs.
Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes.
`--resetTable`
This will delete all the data loaded into transtion_domain. It is helpful if you want to see the entries reload from scratch or for clearing test data.
This will delete all the data in transtion_domain. It is helpful if you want to see the entries reload from scratch or for clearing test data.
### STEP 2: Transfer Transition Domain data into main Domain tables
@ -224,7 +227,7 @@ Now that we've loaded all the data into TransitionDomain, we need to update the
In the same terminal as used in STEP 1, run the command below;
(This will parse the data in TransitionDomain and either create a corresponding Domain object, OR, if a corresponding Domain already exists, it will update that Domain with the incoming status. It will also create DomainInvitation objects for each user associated with the domain):
```shell
docker compose run -T app ./manage.py transfer_transition_domains_to_domains
docker compose run -T app ./manage.py transfer_transition_domains_to_domains --debug
```
**OPTIONAL COMMAND LINE ARGUMENTS**:
@ -232,4 +235,68 @@ docker compose run -T app ./manage.py transfer_transition_domains_to_domains
This will print out additional, detailed logs.
`--limitParse 100`
Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes.
Directs the script to load only the first 100 entries into the table. You can adjust this number as needed for testing purposes.
### STEP 3: Send Domain invitations
### Run the send invitations script
To send invitations for every transition domain in the transition domain table, execute the following command:
`docker compose run -T app send_domain_invitations -s`
### STEP 4: Test the results
### Run the migration analyzer
This script's main function is to scan the transition domain and domain tables for any anomalies. It produces a simple report of missing or duplicate data. NOTE: some missing data might be expected depending on the nature of our migrations so use best judgement when evaluating the results.
**ANALYZE ONLY**
To analyze our database without running migrations, execute the script without any optional arguments:
`docker compose run -T app ./manage.py master_domain_migrations --debug`
**RUN MIGRATIONS FEATURE**
To run the migrations again (all above migration steps) before analyzing, execute the following command (read the documentation on the terminal arguments below. Everything used by the migration scripts can also be passed into this script and will have the same effects). NOTE: --debug and --prompt allow you to step through the migration process and exit it after each step if you need to. It is recommended that you use these arguments when using the --runMigrations feature:
`docker compose run -T app ./manage.py master_domain_migrations --runMigrations --debug --prompt`
#### OPTIONAL ARGUMENTS
`--runMigrations`
A boolean (default to true), which triggers running
all scripts (in sequence) for transition domain migrations
`--migrationDirectory`
**default="migrationData"** (<--This is the sandbox directory)
The location of the files used for load_transition_domain migration script
EXAMPLE USAGE:
--migrationDirectory /app/tmp
`--migrationFilenames`
**default=escrow_domain_contacts.daily.gov.GOV.txt,escrow_contacts.daily.gov.GOV.txt,escrow_domain_statuses.daily.gov.GOV.txt** (<--These are the usual names for the files. The script will throw a warning if it cannot find these exact files, in which case you will need to supply the correct filenames)
The files used for load_transition_domain migration script.
Must appear IN ORDER and comma-delimited:
EXAMPLE USAGE:
--migrationFilenames domain_contacts_filename.txt,contacts_filename.txt,domain_statuses_filename.txt
where...
- domain_contacts_filename is the Data file with domain contact information
- contacts_filename is the Data file with contact information
- domain_statuses_filename is the Data file with domain status information
`--sep`
Delimiter for the migration scripts to correctly parse the given text files.
(usually this can remain at default value of |)
`--debug`
A boolean (default to true), which activates additional print statements
`--prompt`
A boolean (default to true), which activates terminal prompts
that allows the user to step through each portion of this
script.
`--limitParse`
Used by the migration scripts (load_transition_domain) to set the limit for the
number of data entries to insert. Set to 0 (or just don't use this
argument) to parse every entry. This was provided primarily for testing
purposes
`--resetTable`
Used by the migration scripts to trigger a prompt for deleting all table entries.
Useful for testing purposes, but USE WITH CAUTION
```