Add open source documentation about BRDA deposits

Also contains some fixes to the RDE documentation, and adds some site map entries.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136193503

docs/operational-procedures/brda-deposits.md

@@ -0,0 +1,68 @@
+# BRDA Deposits
+
+Bulk Registration Data Access (BRDA) is a backup deposit program mandated by
+ICANN for most gTLD registrars (ccTLDs are not required to provide BRDA
+deposits). Some information related to BRDA can be found at:
+
+https://icannwiki.com/Onboarding_Information_Request#BRDA
+
+BRDA deposits are generated by the
+[RdeStagingAction](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeStagingAction.java)
+job. This is the same job that generates RDE deposits. Its Javadoc goes into
+great detail about how it's implemented.
+
+The [RDE task](./rde-deposits.md) performs BRDA processing at 00:00:00 UTC every
+Tuesday. RDE runs every day, but only performs the following BRDA steps on
+Tuesday (configurable via the `@Config("brdaDayOfWeek")` parameter):
+
+*   Check the BRDA `Cursor`
+*   Create a staging file named:
+    *   `gs://{PROJECT-ID}-rde/TLD_YYYY-MM-DD_thin_S1_R0.xml.ghostryde`
+*   Enqueue a BrdaCopyTask and roll forward the cursor
+
+The BRDA copy task reads the previous file and creates two files:
+
+```
+    gs://{PROJECT-ID}-icann-brda/TLD_YYYY-MM-DD_thin_S1_R0.ryde
+    gs://{PROJECT-ID}-icann-brda/TLD_YYYY-MM-DD_thin_S1_R0.sig
+```
+
+If you see an `xml.ghostryde` file but not the others, an error has occurred
+during the mapreduce process. If you see the files in the
+{PROJECT-ID}-icann-brda bucket as well, the process has completed successfully.
+
+Once the files have been created, they must be stored on an sFTP server from
+which ICANN can pull the files. The Nomulus project does not provide this last
+step; you will need to set up an sFTP server yourself, and copy the files from
+Google Cloud Storage to the server.
+
+The cursor can be checked using the `nomulus pending_escrow` command. files
+available for ICANN via SFTP.
+
+## Generating BRDA deposits manually
+
+*   Get a list of "REAL" (as opposed to TEST) TLDs. Doublecheck that the command
+    output doesn't contain any TLDs for tests.
+
+```shell
+$ registry-tool -e production list_tlds --fields=tldStr,tldType | grep REAL | awk '{print $1}' > realtlds.txt`
+```
+
+*   Generate .ryde and .sig files of TLDs specified for given date(s) in the
+    current directory.
+
+```shell
+$ mkdir /tmp/brda.$$; for date in 2015-02-26 2015-03-05; \
+    do for tld in $(cat realtlds.txt); \
+    do nomulus -e production create_brda_deposit --tld=${tld} --watermark=${date}T00:00:00Z --outdir=/tmp/brda.$$ & sleep 30; \
+    done; \
+    done
+```
+
+*   Store the generated files to the GCS bucket.
+
+```shell
+$ gsutil -m cp /tmp/brda.$$/*.{ryde,sig} gs://{PROJECT-ID}-icann-brda/`
+```
+
+*   Mirror the files in the GCS bucket to the sFTP server.

docs/operational-procedures/rde-deposits.md

@@ -12,7 +12,7 @@ phases:
 1.  [Staging](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeStagingAction.java):
     Generate XML deposit and XML report files on Google Cloud Storage.
 2.  [Upload](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeUploadAction.java):
-    Transmit XML deposit to the escrow provider via SFTP.
+    Transmit XML deposit to the escrow provider via sFTP.
 3.  [Report](https://github.com/google/nomulus/blob/master/java/google/registry/rde/RdeReportAction.java):
     Transmit XML *report* file to ICANN via HTTPS.
 
@@ -21,6 +21,10 @@ When each task succeeds, it automatically enqueues a task for the next phase in
 the process. The staging files are stored in Google Cloud Storage indefinitely,
 encrypted with the GhostRyDE container format.
 
+Note that in order for the automated RDE processing to work correctly, you will
+need to implement a working and secure key store from which RDE can pull the
+private key used to transmit the deposits via sFTP.
+
 For each phase in the process, the system maintains a `Cursor` entity in
 Datastore, which contains a timestamp indicating that everything up to the day
 before the timestamp is current. For RDE, there are four cursor types:
@@ -72,7 +76,7 @@ gs://{PROJECT-ID}-rde/zip_2015-05-16.xml.length
 
 ## Normal launch
 
-Under normal circumstances, RDE is launched by CronFanoutServlet, configured in
+Under normal circumstances, RDE is launched by TldFanoutAction, configured in
 cron.xml. If the App Engine's cron executor isn't working, you can spawn it
 manually by visiting the following URL:
 
@@ -233,8 +237,9 @@ $ ls *.ryde *.sig
 
 ### Verifying the deposit signature (optional)
 
+To verify the deposit signature, you will need a file containing the public key.
+
 ```shell
-$ nomulus -e production get_key --name rde-signing-public > rde-signing-public
 $ (umask 0077; mkdir gpgtemp)
 $ GNUPGHOME=gpgtemp gpg --import ./rde-signing-public
 $ GNUPGHOME=gpgtemp gpg --verify ${tld}_${date}_full_S1_R0-report.{sig,ryde}
@@ -247,13 +252,12 @@ $ rm -rf gpgtemp
 ### Uploading the encrypted deposit and signature files
 
 NOTE: If you need to manually upload files directly to the escrow provider, only
-upload the .ryde and .sig files. DO NOT upload any other files.
+upload the .ryde and .sig files. DO NOT upload any other files. You will need a
+file containing the private key.
 
 ```shell
-# First, you'll need to get the SSH private key from KeyStore
-$ (umask 0077; nomulus -e production get_key --name rde-ssh-client-private > escrow_ssh)
 # Next, sftp to the server
-$ sftp -i escrow_ssh ${user}@${host}:Outbox
+$ sftp -i ./rde-ssh-client-private ${user}@${host}:Outbox
 Connected to ${host}.
 sftp> ls
 # Once in the Outbox/ directory, you can change your local directory to where you have the escrow files
@@ -263,15 +267,15 @@ sftp> put ${tld}_2015-05-16_full_S1_R0.sig
 ```
 
 It would be convenient to have the following in your `~/.ssh/config` file and
-store the SSH private key that you stored in `escrow_ssh` as
+store the SSH private key that you stored in `rde-ssh-client-private` as
-`~/.ssh/id_rsa_escrow` so that you can simply run `$ sftp escrow` to connect to
+`~/.ssh/id_rsa_rde` so that you can simply run `$ sftp rde` to connect to
 the sFTP server.
 
 ```
 Host escrow
   Hostname $host
   User $user
-  IdentityFile ~/.ssh/id_rsa_escrow
+  IdentityFile ~/.ssh/id_rsa_rde
 ```
 
 ## Resending the ICANN notification report