diff --git a/docs/operational-procedures/brda-deposits.md b/docs/operational-procedures/brda-deposits.md new file mode 100644 index 000000000..5305feeac --- /dev/null +++ b/docs/operational-procedures/brda-deposits.md @@ -0,0 +1,68 @@ +# BRDA Deposits + +Bulk Registration Data Access (BRDA) is a backup deposit program mandated by +ICANN for most gTLD registrars (ccTLDs are not required to provide BRDA +deposits). Some information related to BRDA can be found at: + +https://icannwiki.com/Onboarding_Information_Request#BRDA + +BRDA deposits are generated by the +[RdeStagingAction](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeStagingAction.java) +job. This is the same job that generates RDE deposits. Its Javadoc goes into +great detail about how it's implemented. + +The [RDE task](./rde-deposits.md) performs BRDA processing at 00:00:00 UTC every +Tuesday. RDE runs every day, but only performs the following BRDA steps on +Tuesday (configurable via the `@Config("brdaDayOfWeek")` parameter): + +* Check the BRDA `Cursor` +* Create a staging file named: + * `gs://{PROJECT-ID}-rde/TLD_YYYY-MM-DD_thin_S1_R0.xml.ghostryde` +* Enqueue a BrdaCopyTask and roll forward the cursor + +The BRDA copy task reads the previous file and creates two files: + +``` + gs://{PROJECT-ID}-icann-brda/TLD_YYYY-MM-DD_thin_S1_R0.ryde + gs://{PROJECT-ID}-icann-brda/TLD_YYYY-MM-DD_thin_S1_R0.sig +``` + +If you see an `xml.ghostryde` file but not the others, an error has occurred +during the mapreduce process. If you see the files in the +{PROJECT-ID}-icann-brda bucket as well, the process has completed successfully. + +Once the files have been created, they must be stored on an sFTP server from +which ICANN can pull the files. The Nomulus project does not provide this last +step; you will need to set up an sFTP server yourself, and copy the files from +Google Cloud Storage to the server. + +The cursor can be checked using the `nomulus pending_escrow` command. files +available for ICANN via SFTP. + +## Generating BRDA deposits manually + +* Get a list of "REAL" (as opposed to TEST) TLDs. Doublecheck that the command + output doesn't contain any TLDs for tests. + +```shell +$ registry-tool -e production list_tlds --fields=tldStr,tldType | grep REAL | awk '{print $1}' > realtlds.txt` +``` + +* Generate .ryde and .sig files of TLDs specified for given date(s) in the + current directory. + +```shell +$ mkdir /tmp/brda.$$; for date in 2015-02-26 2015-03-05; \ + do for tld in $(cat realtlds.txt); \ + do nomulus -e production create_brda_deposit --tld=${tld} --watermark=${date}T00:00:00Z --outdir=/tmp/brda.$$ & sleep 30; \ + done; \ + done +``` + +* Store the generated files to the GCS bucket. + +```shell +$ gsutil -m cp /tmp/brda.$$/*.{ryde,sig} gs://{PROJECT-ID}-icann-brda/` +``` + +* Mirror the files in the GCS bucket to the sFTP server. diff --git a/docs/operational-procedures/rde-deposits.md b/docs/operational-procedures/rde-deposits.md index 30dabab54..593b39032 100644 --- a/docs/operational-procedures/rde-deposits.md +++ b/docs/operational-procedures/rde-deposits.md @@ -12,7 +12,7 @@ phases: 1. [Staging](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeStagingAction.java): Generate XML deposit and XML report files on Google Cloud Storage. 2. [Upload](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeUploadAction.java): - Transmit XML deposit to the escrow provider via SFTP. + Transmit XML deposit to the escrow provider via sFTP. 3. [Report](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeReportAction.java): Transmit XML *report* file to ICANN via HTTPS. @@ -21,6 +21,10 @@ When each task succeeds, it automatically enqueues a task for the next phase in the process. The staging files are stored in Google Cloud Storage indefinitely, encrypted with the GhostRyDE container format. +Note that in order for the automated RDE processing to work correctly, you will +need to implement a working and secure key store from which RDE can pull the +private key used to transmit the deposits via sFTP. + For each phase in the process, the system maintains a `Cursor` entity in Datastore, which contains a timestamp indicating that everything up to the day before the timestamp is current. For RDE, there are four cursor types: @@ -72,7 +76,7 @@ gs://{PROJECT-ID}-rde/zip_2015-05-16.xml.length ## Normal launch -Under normal circumstances, RDE is launched by CronFanoutServlet, configured in +Under normal circumstances, RDE is launched by TldFanoutAction, configured in cron.xml. If the App Engine's cron executor isn't working, you can spawn it manually by visiting the following URL: @@ -233,8 +237,9 @@ $ ls *.ryde *.sig ### Verifying the deposit signature (optional) +To verify the deposit signature, you will need a file containing the public key. + ```shell -$ nomulus -e production get_key --name rde-signing-public > rde-signing-public $ (umask 0077; mkdir gpgtemp) $ GNUPGHOME=gpgtemp gpg --import ./rde-signing-public $ GNUPGHOME=gpgtemp gpg --verify ${tld}_${date}_full_S1_R0-report.{sig,ryde} @@ -247,13 +252,12 @@ $ rm -rf gpgtemp ### Uploading the encrypted deposit and signature files NOTE: If you need to manually upload files directly to the escrow provider, only -upload the .ryde and .sig files. DO NOT upload any other files. +upload the .ryde and .sig files. DO NOT upload any other files. You will need a +file containing the private key. ```shell -# First, you'll need to get the SSH private key from KeyStore -$ (umask 0077; nomulus -e production get_key --name rde-ssh-client-private > escrow_ssh) # Next, sftp to the server -$ sftp -i escrow_ssh ${user}@${host}:Outbox +$ sftp -i ./rde-ssh-client-private ${user}@${host}:Outbox Connected to ${host}. sftp> ls # Once in the Outbox/ directory, you can change your local directory to where you have the escrow files @@ -263,15 +267,15 @@ sftp> put ${tld}_2015-05-16_full_S1_R0.sig ``` It would be convenient to have the following in your `~/.ssh/config` file and -store the SSH private key that you stored in `escrow_ssh` as -`~/.ssh/id_rsa_escrow` so that you can simply run `$ sftp escrow` to connect to +store the SSH private key that you stored in `rde-ssh-client-private` as +`~/.ssh/id_rsa_rde` so that you can simply run `$ sftp rde` to connect to the sFTP server. ``` Host escrow   Hostname $host   User $user -  IdentityFile ~/.ssh/id_rsa_escrow +  IdentityFile ~/.ssh/id_rsa_rde ``` ## Resending the ICANN notification report