zydronium/manage.get.gov
1
0
Fork 0
mirror of https://github.com/cisagov/manage.get.gov.git synced 2025-06-29 07:43:32 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'nl/981-test-domain-migration-script' into za/additional-data-transferred-domains

Browse source
This commit is contained in:
zandercymatics 2023-11-06 09:31:16 -07:00
commit ed791825e4
No known key found for this signature in database
GPG key ID: FF4636ABEC9682B7
5 changed files with 36 additions and 83 deletions
Download patch file Download diff file Expand all files Collapse all files

27
docs/operations/data_migration.md
View file

@ -92,15 +92,15 @@ We can do this both locally and in a sandbox.
Load migration data onto a production or sandbox environment
**WARNING:** All files uploaded in this manner are temporary, i.e. they will be deleted when the app is restaged.
Do not use this method to store data you want to keep around permanently.
Do not use these environments to store data you want to keep around permanently. We don't want sensitive data to be accidentally present in our application environments.
#### STEP 1: Using cat to transfer data to sandboxes
```bash
cat {LOCAL_PATH_TO_FILE} | cf ssh {FULL_NAME_OF_YOUR_SANDBOX_HERE} -c "cat > /home/vcap/tmp/{DESIRED_NAME_OF_FILE}"
cat {LOCAL_PATH_TO_FILE} | cf ssh {APP_NAME_IN_ENVIRONMENT} -c "cat > /home/vcap/tmp/{DESIRED_NAME_OF_FILE}"
```
* FULL_NAME_OF_YOUR_SANDBOX_HERE - Name of your sandbox, ex: getgov-za
* APP_NAME_IN_ENVIRONMENT - Name of the app running in your environment, e.g. getgov-za or getgov-stable
* LOCAL_PATH_TO_FILE - Path to the file you want to copy, ex: src/tmp/escrow_contacts.daily.gov.GOV.txt
* DESIRED_NAME_OF_FILE - Use this to specify the filename and type, ex: test.txt or escrow_contacts.daily.gov.GOV.txt
@ -122,25 +122,23 @@ cf login -a api.fr.cloud.gov --sso
##### Target your workspace
```bash
cf target -o cisa-dotgov -s {SANDBOX_NAME}
cf target -o cisa-dotgov -s {ENVIRONMENT_NAME}
```
*SANDBOX_NAME* - Name of your sandbox, ex: za or ab
*ENVIRONMENT_NAME* - Name of your sandbox, ex: za or ab
##### Run the scp command
Use the following command to transfer the desired file:
```shell
scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app {FULL_NAME_OF_YOUR_SANDBOX_HERE} --guid)/processes | jq -r '.resources[]
scp -P 2222 -o User=cf:$(cf curl /v3/apps/$(cf app {APP_NAME_IN_ENVIRONMENT} --guid)/processes | jq -r '.resources[]
| select(.type=="web") | .guid')/0 {LOCAL_PATH_TO_FILE} ssh.fr.cloud.gov:tmp/{DESIRED_NAME_OF_FILE}
```
The items in curly braces are the values that you will manually replace.
These are as follows:
* FULL_NAME_OF_YOUR_SANDBOX_HERE - Name of your sandbox, ex: getgov-za
* APP_NAME_IN_ENVIRONMENT - Name of the app running in your environment, e.g. getgov-za or getgov-stable
* LOCAL_PATH_TO_FILE - Path to the file you want to copy, ex: src/tmp/escrow_contacts.daily.gov.GOV.txt
* DESIRED_NAME_OF_FILE - Use this to specify the filename and type, ex: test.txt or escrow_contacts.daily.gov.GOV.txt
NOTE: If you'd wish to change what directory these files are uploaded to, you can change `ssh.fr.cloud.gov:tmp/` to `ssh.fr.cloud.gov:{DIRECTORY_YOU_WANT}/`, but be aware that this makes data migration more tricky than it has to be.
##### Get a temp auth code
The scp command requires a temporary authentication code. Open a new terminal instance (while keeping the current one open),
@ -158,7 +156,7 @@ Due to the nature of how Cloud.gov operates, the getgov directory is dynamically
##### SSH into your sandbox
```shell
cf ssh {FULL_NAME_OF_YOUR_SANDBOX_HERE}
cf ssh {APP_NAME_IN_ENVIRONMENT}
```
##### Open a shell
@ -196,7 +194,7 @@ cat ../tmp/{filename} > migrationdata/{filename}
```
*You are now ready to run migration scripts (see "Running the Migration Scripts")*
*You are now ready to run migration scripts (see [Running the Migration Scripts](running-the-migration-scripts))*
### SECTION 2 - LOCAL MIGRATION SETUP (TESTING PURPOSES ONLY)
@ -205,12 +203,11 @@ cat ../tmp/{filename} > migrationdata/{filename}
In order to run the scripts locally, we need to add the files to a folder under `src/`.
This will allow Docker to mount the files to a container (under `/app`) for our use.
- Create a folder called `tmp` underneath `src/`
- Add the above files to this folder
- Add the above files to the `migrationdata/` folder
- Open a terminal and navigate to `src/`
*You are now ready to run migration scripts (see "Running the Migration Scripts")*
*You are now ready to run migration scripts.*
## Transition Domains (Part 2) - Running the Migration Scripts
@ -218,7 +215,7 @@ This will allow Docker to mount the files to a container (under `/app`) for our
### STEP 1: Load Transition Domains
Run the following command, making sure the filepaths point to the right location. This will parse the three given files and load the information into the TransitionDomain table. (NOTE: If working in the sandbox, change "/app/tmp" to point to the sandbox directory)
Run the following command, making sure the file paths point to the right location. This will parse the three given files and load the information into the TransitionDomain table. (NOTE: If working in cloud.gov, change "/app/tmp" to point to the `migrationdata/` directory)
```shell
docker compose run -T app ./manage.py load_transition_domain /app/tmp/escrow_domain_contacts.daily.gov.GOV.txt /app/tmp/escrow_contacts.daily.gov.GOV.txt /app/tmp/escrow_domain_statuses.daily.gov.GOV.txt --debug
```

62
src/registrar/management/commands/cat_files_into_getgov.py
View file

@ -3,7 +3,7 @@ import glob
import logging
import os
import string
import shutil
from django.core.management import BaseCommand
@ -36,65 +36,21 @@ class Command(BaseCommand):
if not file_extension or not isinstance(file_extension, str):
raise ValueError(f"Invalid file extension '{file_extension}'")
matching_extensions = glob.glob(f"../tmp/*.{file_extension}")
if not matching_extensions:
matching_files = glob.glob(f"../tmp/*.{file_extension}")
if not matching_files:
logger.error(f"No files with the extension {file_extension} found")
return None
for src_file_path in matching_extensions:
for src_file_path in matching_files:
filename = os.path.basename(src_file_path)
exit_status = -1
do_command = True
desired_file_path = f"{directory}/{filename}"
desired_file_path = os.path.join(directory, filename)
if os.path.exists(desired_file_path):
# For linter
prompt = "Do you want to replace it?"
replace = f"{desired_file_path} already exists. {prompt}"
if not helper.query_yes_no(replace):
do_command = False
continue
try:
if do_command:
copy_from = f"../tmp/{filename}"
exit_status = self.cat(copy_from, desired_file_path)
except ValueError as err:
raise err
finally:
if exit_status == 0:
logger.info(f"Successfully copied {filename}")
else:
logger.error(f"Failed to copy {filename}")
def cat(self, copy_from, copy_to):
"""Runs the cat command to
copy_from a location to copy_to a location"""
# copy_from will be system defined
self.check_file_path(copy_from, check_directory=False)
self.check_file_path(copy_to)
# This command can only be ran from inside cf ssh getgov-{sandbox}
# It has no utility when running locally, and to exploit this
# you would have to have ssh access anyway, which is a bigger problem.
exit_status = os.system(f"cat {copy_from} > {copy_to}") # nosec
return exit_status
def check_file_path(self, file_path: str, check_directory=True):
"""Does a check on user input to ensure validity"""
if not isinstance(file_path, str):
raise ValueError("Invalid path provided")
# Remove any initial/final whitespace
file_path = file_path.strip()
# Check for any attempts to move up in the directory structure
if ".." in file_path and check_directory:
raise ValueError("Moving up in the directory structure is not allowed")
# Check for any invalid characters
valid_chars = f"/-_.() {string.ascii_letters}{string.digits}"
for char in file_path:
if char not in valid_chars:
raise ValueError(f"Invalid character {char} in file path")
return file_path
src_file_path = f"../tmp/{filename}"
shutil.copy(src_file_path, desired_file_path)

14
src/registrar/tests/data/test_contacts.txt
View file

@ -1,7 +1,7 @@
TESTUSER|52563_CONTACT_GOV-VRSN|919-000-0000||918-000-0000||testuser@gmail.com|GSA|VERISIGN|ctldbatch|2021-06-30T17:58:09Z|VERISIGN|ctldbatch|2021-06-30T18:18:09Z|
RJD1|52545_CONTACT_GOV-VRSN|919-000-0000||918-000-0000||agustina.wyman7@test.com|GSA|VERISIGN|ctldbatch|2021-06-29T18:53:09Z|VERISIGN|ctldbatch|2021-06-29T18:58:08Z|
JAKING|52555_CONTACT_GOV-VRSN|919-000-0000||918-000-0000||susy.martin4@test.com|GSA|VERISIGN|ctldbatch|2021-06-30T15:23:10Z|VERISIGN|ctldbatch|2021-06-30T15:38:10Z|
JBOONE|52556_CONTACT_GOV-VRSN|919-000-0000||918-000-0000||stephania.winters4@test.com|GSA|VERISIGN|ctldbatch|2021-06-30T15:23:10Z|VERISIGN|ctldbatch|2021-06-30T18:28:09Z|
MKELLEY|52557_CONTACT_GOV-VRSN|919-000-0000||918-000-0000||alexandra.bobbitt5@test.com|GSA|VERISIGN|ctldbatch|2021-06-30T15:23:10Z|VERISIGN|ctldbatch|2021-08-02T22:13:09Z|
CWILSON|52562_CONTACT_GOV-VRSN|919-000-0000||918-000-0000||jospeh.mcdowell3@test.com|GSA|VERISIGN|ctldbatch|2021-06-30T17:58:09Z|VERISIGN|ctldbatch|2021-06-30T18:33:09Z|
LMCCADE|52563_CONTACT_GOV-VRSN|919-000-0000||918-000-0000||reginald.ratcliff4@test.com|GSA|VERISIGN|ctldbatch|2021-06-30T17:58:09Z|VERISIGN|ctldbatch|2021-06-30T18:18:09Z|
TESTUSER|12363_CONTACT|123-123-1234||918-000-0000||testuser@gmail.com|GSA|SOMECOMPANY|ctldbatch|2021-06-30T17:58:09Z|SOMECOMPANY|ctldbatch|2021-06-30T18:18:09Z|
USER1|12345_CONTACT|123-123-1234||918-000-0000||agustina.wyman7@test.com|GSA|SOMECOMPANY|ctldbatch|2021-06-29T18:53:09Z|SOMECOMPANY|ctldbatch|2021-06-29T18:58:08Z|
USER2|12355_CONTACT|123-123-1234||918-000-0000||susy.martin4@test.com|GSA|SOMECOMPANY|ctldbatch|2021-06-30T15:23:10Z|SOMECOMPANY|ctldbatch|2021-06-30T15:38:10Z|
USER3|12356_CONTACT|123-123-1234||918-000-0000||stephania.winters4@test.com|GSA|SOMECOMPANY|ctldbatch|2021-06-30T15:23:10Z|SOMECOMPANY|ctldbatch|2021-06-30T18:28:09Z|
USER4|12357_CONTACT|123-123-1234||918-000-0000||alexandra.bobbitt5@test.com|GSA|SOMECOMPANY|ctldbatch|2021-06-30T15:23:10Z|SOMECOMPANY|ctldbatch|2021-08-02T22:13:09Z|
USER5|12362_CONTACT|123-123-1234||918-000-0000||jospeh.mcdowell3@test.com|GSA|SOMECOMPANY|ctldbatch|2021-06-30T17:58:09Z|SOMECOMPANY|ctldbatch|2021-06-30T18:33:09Z|
USER6|12363_CONTACT|123-123-1234||918-000-0000||reginald.ratcliff4@test.com|GSA|SOMECOMPANY|ctldbatch|2021-06-30T17:58:09Z|SOMECOMPANY|ctldbatch|2021-06-30T18:18:09Z|

12
src/registrar/tests/data/test_domain_contacts.txt
View file

@ -1,8 +1,8 @@
Anomaly.gov|ANOMALY|tech
TestDomain.gov|TESTUSER|admin
NEHRP.GOV|RJD1|admin
NEHRP.GOV|JAKING|tech
NEHRP.GOV|JBOONE|billing
NELSONCOUNTY-VA.GOV|MKELLEY|admin
NELSONCOUNTY-VA.GOV|CWILSON|billing
NELSONCOUNTY-VA.GOV|LMCCADE|tech
FakeWebsite1|USER1|admin
FakeWebsite1|USER2|tech
FakeWebsite1|USER3|billing
FakeWebsite2.GOV|USER4|admin
FakeWebsite2.GOV|USER5|billing
FakeWebsite2.GOV|USER6|tech

4
src/registrar/tests/data/test_domain_statuses.txt
View file

@ -1,4 +1,4 @@
Anomaly.gov|muahaha|
TestDomain.gov|ok|
NEHRP.GOV|serverHold|
NELSONCOUNTY-VA.GOV|Hold|
FakeWebsite1.GOV|serverHold|
FakeWebsite2.GOV|Hold|