google-nomulus/docs/operational-procedures/rde-deposits.md

RDE Deposits

Overview

Many registries are required by ICANN to make daily escrow deposits of the registry data. Note that this applies mainly to gTLDs. ccTLDs do not generally have this requirement, in which case the following would not apply.

The RDE process takes care of escrow deposit processing. It happens in three phases:

1. Staging: Generate XML deposit and XML report files on Google Cloud Storage.
2. Upload: Transmit XML deposit to the escrow provider via sFTP.
3. Report: Transmit XML report file to ICANN via HTTPS.

Each phase happens with an App Engine task queue entry that retries on failure. When each task succeeds, it automatically enqueues a task for the next phase in the process. The staging files are stored in Google Cloud Storage indefinitely, encrypted with the GhostRyDE container format.

Note that in order for the automated RDE processing to work correctly, you will need to implement a working and secure key store from which RDE can pull the private key used to transmit the deposits via sFTP.

For each phase in the process, the system maintains a Cursor entity in Datastore, which contains a timestamp indicating that everything up to the day before the timestamp is current. For RDE, there are four cursor types:

• RDE_STAGING
• RDE_UPLOAD
• RDE_UPLOAD_SFTP
• RDE_REPORT

To view the cursors, you can use the nomulus list_cursors command. Here are some examples:

$ nomulus -e production list_cursors --type RDE_STAGING
2015-05-22T00:00:00.000Z tld1
2015-05-22T00:00:00.000Z tld2
2015-05-22T00:00:00.000Z tld3
2015-05-22T00:00:00.000Z tld4
[....]
$ nomulus -e production list_cursors --type RDE_UPLOAD
2015-05-22T00:00:00.000Z tld1
2015-05-22T00:00:00.000Z tld2
2015-05-22T00:00:00.000Z tld3
2015-05-22T00:00:00.000Z tld4
[...]
$ nomulus -e production list_cursors --type RDE_REPORT
2015-05-22T00:00:00.000Z tld1
2015-05-22T00:00:00.000Z tld2
2015-05-22T00:00:00.000Z tld3
2015-05-22T00:00:00.000Z tld4
[....]

The RDE_UPLOAD_SFTP cursor is used to avoid uploading multiple files to the escrow provider within too short a time span, as described below.

Listing deposits in Cloud Storage

You can list the files in Cloud Storage for a given TLD using the gsutil tool. All files are stored in the {PROJECT-ID}-rde bucket, where {PROJECT-ID} is the name of the App Engine project for the particular environment you are checking.

$ gsutil ls gs://{PROJECT-ID}-rde/zip_2015-05-16*
gs://{PROJECT-ID}-rde/zip_2015-05-16-report.xml.ghostryde
gs://{PROJECT-ID}-rde/zip_2015-05-16.xml.ghostryde
gs://{PROJECT-ID}-rde/zip_2015-05-16.xml.length

Normal launch

Under normal circumstances, RDE is launched by TldFanoutAction, configured in cron.xml. If the App Engine's cron executor isn't working, you can spawn it manually by visiting the following URL:

https://backend-dot-{PROJECT-ID}.appspot.com/_dr/task/rdeStaging

That will spawn a staging task for each TLD under the backend module in that App Engine project.

Notification of upload problems

If there is a problem with the deposit from the escrow provider's side, they will usually email a notification. Here is an example of such a notification from Iron Mountain, one common escrow provider:

Greetings,

The Registry Data Escrow deposit for XXX has failed the verification process.
Details can be found below.

------------- Name Check ---------------- 
Name Check Filename : xxx_2015-05-16_full_S1_R0.sig 
Name Check : SUCCESSFUL 
Name Check Filename : xxx_2015-05-16_full_S1_R0.ryde 
Name Check : SUCCESSFUL 
------------- Signature Check ---------------- 
Signature Check Filename : xxx_2015-05-16_full_S1_R0.ryde 
Digital Signature Check : FAILED 

Thank you

Iron Mountain

Iron Mountain-specific caveat: never re-upload between 10 p.m. and 3 a.m. UTC

When Iron Mountain receives two deposits for the same TLD within two hours, it merges them together. So when resending deposits, don't do it anywhere near midnight UTC, since that might get merged with the output of the regular cron job. And if you need to resend deposits for more than one day on a single TLD, space the uploading at least two hours apart.

Note that this warning only applies when you (re)upload files directly to the sFTP server. There's an RDE_UPLOAD_SFTP cursor that prevents the production system from uploading twice in a two hour window, so when you let the production job upload missing deposits, it will be safe. Therefore, one safe approach is to reset the cursor, then kick off the production job manually.

Decrypting GhostRyDE files

Sometimes you'll want to take a peek at the contents of a deposit that's been staged to cloud storage. Use this command:

$ gsutil cat gs://{PROJECT-ID}-rde/foo.ghostryde | nomulus -e production ghostryde --decrypt | less

Identifying which phase of the process failed

Analyze the GAE logs on the backend module.

If the rdeStaging task failed, then it's likely the files do not exist in cloud storage.

If the rdeUpload task failed, then it's likely the files were not uploaded to the escrow provider's sFTP server.

If the rdeReport task failed, then it's likely the report file was not sent to ICANN.

Generating RDE files

You can use the nomulus command to generate the RDE files on a per-TLD basis. Here's an example:

# Define the tld/date combination.

$ tld=xxx
$ date=2015-05-16

# 1. Generate the deposit. This drops a bunch of files in the current directory.

$ nomulus -e production generate_escrow_deposit --tld=${tld} --watermark=${date}T00:00:00Z

$ ls -l
total 22252
-rw-r----- 1 tjb eng     2292 May 19 16:49 xxx_2015-05-16_full_S1_R0-report.xml
-rw-r----- 1 tjb eng  1321343 May 19 16:49 xxx_2015-05-16_full_S1_R0.ryde
-rw-r----- 1 tjb eng      361 May 19 16:49 xxx_2015-05-16_full_S1_R0.sig
-rw-r----- 1 tjb eng 21448005 May 19 16:49 xxx_2015-05-16_full_S1_R0.xml
-rw-r----- 1 tjb eng      977 May 19 16:49 xxx.pub

$ nomulus -e production validate_escrow_deposit -i ${tld}_${date}_full_S1_R0.xml
ID: AAAACTK2AU2AA
Previous ID: null
Type: FULL
Watermark: 2015-05-16T00:00:00.000Z
RDE Version: 1.0
RDE Object URIs:
  - urn:ietf:params:xml:ns:rdeContact-1.0
  - urn:ietf:params:xml:ns:rdeDomain-1.0
  - urn:ietf:params:xml:ns:rdeHeader-1.0
  - urn:ietf:params:xml:ns:rdeHost-1.0
  - urn:ietf:params:xml:ns:rdeRegistrar-1.0
Contents:
  - XjcRdeContact: 4,224 entries
  - XjcRdeDomain: 2,667 entries
  - XjcRdeHeader: 1 entry
  - XjcRdeHost: 35,932 entries
  - XjcRdeRegistrar: 146 entries
RDE deposit is XML schema valid

# 2. GhostRyDE it!

$ nomulus -e production ghostryde --encrypt \
    --input=${tld}_${date}_full_S1_R0-report.xml \
    --output=${tld}_${date}_full_S1_R0-report.xml.ghostryde

$ nomulus -e production ghostryde --encrypt \
    --input=${tld}_${date}_full_S1_R0.xml \
    --output=${tld}_${date}_full_S1_R0.xml.ghostryde

# 3. Copy to Cloud Storage so RdeUploadTask can find them.

$ gsutil cp ${tld}_${date}_full_S1_R0{,-report}.xml.ghostryde gs://{PROJECT-ID}-rde/

Updating an RDE cursor

If the rdeStaging or rdeUpload tasks are failing and you find yourself needing to perform this process manually, you must make sure to update the appropriate cursors. Again, you can use the nomulus to perform this operation.

$ nomulus -e production update_cursors --timestamp=2015-05-21T00:00:00Z --type=RDE_STAGING $tld
$ nomulus -e production update_cursors --timestamp=2015-05-21T00:00:00Z --type=RDE_UPLOAD $tld
$ nomulus -e production update_cursors --timestamp=2015-05-21T00:00:00Z --type=RDE_REPORT $tld

Manually uploading files to the escrow provider

These instructions work for Iron Mountain, and should be applicable to other escrow providers as well. We upload the RDE deposits to an sFTP server (see the ConfigModule for specific URLs). First, you need a generated deposit .xml file (see above for how to generate such a file locally, or how to decrypt a .ghostryde file into the .xml original).

Encrypting the RDE deposits for sending to the escrow provider

$ nomulus -e production encrypt_escrow_deposit --tld=$tld --input=${tld}_${date}_full_S1_R0-report.xml
$ ls *.ryde *.sig
  ${tld}_${date}_full_S1_R0-report.ryde
  ${tld}_${date}_full_S1_R0-report.sig

Verifying the deposit signature (optional)

To verify the deposit signature, you will need a file containing the public key.

$ (umask 0077; mkdir gpgtemp)
$ GNUPGHOME=gpgtemp gpg --import ./rde-signing-public
$ GNUPGHOME=gpgtemp gpg --verify ${tld}_${date}_full_S1_R0-report.{sig,ryde}
...
gpg: Good signature from ...
...
$ rm -rf gpgtemp

Uploading the encrypted deposit and signature files

NOTE: If you need to manually upload files directly to the escrow provider, only upload the .ryde and .sig files. DO NOT upload any other files. You will need a file containing the private key.

# Next, sftp to the server
$ sftp -i ./rde-ssh-client-private ${user}@${host}:Outbox
Connected to ${host}.
sftp> ls
# Once in the Outbox/ directory, you can change your local directory to where you have the escrow files
sftp> lcd /path/to/escrow/files/on/local/machine
sftp> put ${tld}_2015-05-16_full_S1_R0.ryde
sftp> put ${tld}_2015-05-16_full_S1_R0.sig

It would be convenient to have the following in your ~/.ssh/config file and store the SSH private key that you stored in rde-ssh-client-private as ~/.ssh/id_rsa_rde so that you can simply run $ sftp rde to connect to the sFTP server.

Host escrow
  Hostname $host
  User $user
  IdentityFile ~/.ssh/id_rsa_rde

Resending the ICANN notification report

As with everything else related to RDE, you can use the nomulus tool to send a notification report to ICANN:

# This command assumes the report XML file is in your current working directory
$ nomulus -e production send_escrow_report_to_icann xxx_2015-05-16_full_S1_R0-report.xml