google-nomulus/docs/operational-procedures.md

# Operational procedures

This document covers procedures that are typically used when running a
production registry system.

## OT&E onboarding

## Managing static premium price lists

### Premium list file format

Nomulus comes with a `StaticPremiumListPricingEngine` that determines premium
prices of domain labels (i.e. the part of the domain name without the TLD) by
checking for their presence on a list of prices in Datastore. `nomulus` is used
to load and update these lists from flat text files. The format of this list is
simple: It is a newline-delimited CSV text file with each line containing the
label and its price (including currency specifier in ISO-4217 format). As an
example:

```
premium,USD 100
superpremium,USD 500
```

By convention, premium lists are named after the TLD they apply to (with one
premium list per TLD). If the above example applied to the `exampletld` example,
then the file would be saved as `exampletld.txt` and the domain name
`premium.exampletld` would cost $100. It is also possible to have a single
premium list that is used on multiple TLDs, for which a different naming
convention would be used.

It is recommended that you store your premium list .txt files in a backed-up
location that is accessible to all members of your team (ideally in a source
control system for revision tracking). These files should be thought of as the
canonical versions of your premium lists. Note that there is no provided way to
reconstruct a premium list .txt file from the premium list that is loaded into
Datastore (though in principle it would be easy to do by writing a tool to do
so), so don't lose those .txt files.

### Creating a premium list

Once the file containing the premium prices is ready, run the
`create_premium_list` command to load it into Datastore as follows:

```shell
$ nomulus -e {ENVIRONMENT} create_premium_list -n exampletld -i exampletld.txt

You are about to save the premium list exampletld with 2 items:
Perform this command? (y/N): y
Successfully saved premium list exampletld
```

`-n` is the name of the list to be created, and `-i` is the input filename. Note
that the convention of naming premium lists after the TLD they are intended to
be used for to is enforced unless the override parameter `-o` is passed, which
allows premium lists to be created with any name.

### Updating a premium list

If the premium list already exists and you want to update it with new prices
from a text file, the procedure is exactly the same, except using the
`update_premium_list` command as follows:

```shell
$ nomulus -e {ENVIRONMENT} update_premium_list -n exampletld -i exampletld.txt

You are about to save the premium list exampletld with 2 items:
Perform this command? (y/N): y
Successfully saved premium list exampletld
```

### Applying a premium list to a TLD

Separate from the management of the contents of individual premium lists, a
premium list must first be applied to a TLD before it will take effect. You will
only need to do this when first creating a premium list; once it has been
applied, it stays applied, and updates to the list are effective automatically.
Note that each TLD can have no more than one premium list applied to it. To
apply a premium list to a TLD, run the `update_tld` command with the following
parameter:

```shell
$ nomulus -e {ENVIRONMENT} update_tld exampletld --premium_list exampletld
Update Registry@exampletld
premiumList -> [null, Key<?>(EntityGroupRoot("cross-tld")/PremiumList("exampletld"))]

Perform this command? (y/N): y
Updated 1 entities.
```

### Checking which premium list is applied to a TLD

The `get_tld` command shows which premium list is applied to a TLD (along with
all other information about a TLD). It is used as follows:

```shell
$ nomulus -e {ENVIRONMENT} get_tld exampletld
[ ... snip ... ]
premiumList=Key<?>(EntityGroupRoot("cross-tld")/PremiumList("exampletld"))
[ ... snip ... ]
```

### Listing all available premium lists

The `list_premium_lists` command is used to list all premium lists in Datastore.
It takes no arguments and displays a simple list of premium lists as follows:

```shell
$ nomulus -e {ENVIRONMENT} list_premium_lists
exampletld
someotherlist
```

## Managing reserved lists

Reserved lists are static lists of labels that are blocked from being registered
for various reasons, usually because of potential abuse.

### Reserved list file format

Reserved lists are handled in a similar way to premium lists (see above), except
that rather than each label having a price, it has a reservation type. The valid
values for reservation types are:

*   **`UNRESERVED`** - The default value for any label that isn't reserved.
    Labels that aren't explictly under any other status implictly have this
    value.
*   **`ALLOWED_IN_SUNRISE`** - The label can be registered during the sunrise
    period by a registrant with a valid claim but it is reserved thereafter.
*   **`MISTAKEN_PREMIUM`** - The label is reserved because it was mistakenly put
    on a premium list. It may be registered during sunrise by a registrant with
    a valid claim but is reserved thereafter.
*   **`RESERVED_FOR_ANCHOR_TENANT`** - The label is reserved for the use of an
    anchor tenant, and can only be registered by someone sending along the EPP
    passcode specified here at time of registration.
*   **`NAME_COLLISION`** - The label is reserved because it is on an [ICANN
    collision
    list](https://www.icann.org/resources/pages/name-collision-2013-12-06-en).
    It may be registered during sunrise by a registrant with a valid claim but
    is reserved thereafter.
*   **`FULLY_BLOCKED`** - The label is fully reserved, no further reason
    specified.

The reservation types are listed in order of increasing precedence, so if a
label is included on the same list multiple lists, or on different lists that
are applied to a single TLD, whichever reservation type is later in the list
takes precedence. E.g. a label being fully blocked in one list always supersedes
it being allowed in sunrise from another list. In general `FULLY_BLOCKED` is by
far the most widely used reservation type for typical TLD use cases.

Here's an example of a small reserved list. Note that
`RESERVED_FOR_ANCHOR_TENANT` is the only reservation type that has a third entry
on the line, that entry being the EPP passcode required to register the domain
(`hunter2` in this case):

```
reserveddomain,FULLY_BLOCKED
availableinga,ALLOWED_IN_SUNRISE
fourletterword,FULLY_BLOCKED
acmecorp,RESERVED_FOR_ANCHOR_TENANT,hunter2
```

There are two types of reserved lists: Those that are intended to apply to a
specifc TLD, and are thus prefixed with the name of the TLD followed by an
underscore, and those that can be applied to any TLD, and are prefixed with
`common_`. For example, a list of reserved labels on the TLD `exampletld` might
be named `exampletld_blocked-names.txt`, whereas a similar list intended to
apply to multiple TLDs might be named `common_blocked-names.txt`. Note that
these naming conventions are enforced by the tooling used to create and apply
reserved lists (see subsequent sections).

### Creating a reserved list

Once the file containing the list of reserved terms is created, run the
`create_reserved_list` command to load it into Datastore as follows. For the
purposes of this example, we are creating a common reserved list named
"common_bad-words".

```shell
$ nomulus -e {ENVIRONMENT} create_reserved_list -n common_bad-words \
    -i common_bad-words.txt
[ ... snip long confirmation prompt ... ]
Perform this command? (y/N): y
Updated 1 entities.
```

`-n` is the name of the reserved list to create, and `-i` is the input file
containing the list. Note that if `-n` is omitted, the list name is inferred
from the input of the filename minus its file extension. It is recommended to
store all lists such that the filename and list name are identical.

### Updating a reserved list

To update the contents of an existing reserved list, make changes to the .txt
file containing the reserved list entries, then pass it as input to the
`update_reserved_list` command as follows:

```shell
$ nomulus -e {ENVIRONMENT} update_reserved_list -n common_bad-words \
    -i common_bad-words.txt
[ ... snip diff of changes to list entries ... ]
Perform this command? (y/N): y
Updated 1 entities.
```

Note that, like the create command, the name of the list is inferred from the
filename if the `-n` parameter is omitted.

### Applying a reserved list to a TLD

Separate from the management of the contents of individual reserved lists,
reserved lists must also be applied to a TLD. Unlike premium lists, for which
each TLD may only have a single list applied, a TLD can have any number of
reserved lists applied. The list of reserved labels for a TLD is the union of
all applied reserved lists, using the precedence rules described earlier when a
label appears in more than one list.

To add a reserved list to a TLD, run the `update_tld` command with the following
parameter:

```shell
$ nomulus -e {ENVIRONMENT} update_tld exampletld \
    --add_reserved_lists common_bad-words
Update Registry@exampletld
reservedLists -> [null, [Key<?>(EntityGroupRoot("cross-tld")/ReservedList("common_bad-words"))]]
Perform this command? (y/N): y
Updated 1 entities.
```

The `--add_reserved_lists` parameter can take a comma-delimited list of reserved
list names if you are applying multiple reserved lists to a TLD. There is also a
`--remove_reserved_lists` parameter that functions as you might expect.

Naming rules are enforced: reserved lists that start with `common_` can be
applied to any TLD (though they don't automatically apply to all TLDs), whereas
reserved lists that start with the name of a TLD can only be applied to the TLD
with that name.

### Checking which reserved lists are applied to a TLD

The `get_tld` command shows which reserved lists (if any) are applied to a TLD,
along with lots of other information about that TLD which is not relevant to our
purposes here. It is used as follows:

```shell
$ nomulus -e {ENVIRONMENT} get_tld exampletld
[ ... snip ... ]
reservedLists=[Key<?>(EntityGroupRoot("cross-tld")/ReservedList("common_bad-words"))]
[ ... snip ... ]
```

### Listing all available reserved lists

The `list_reserved_lists` command is used to list all reserved lists in
Datastore. It takes no arguments and displays a simple list of reserved lists in
newline-delimited format as follows:

```shell
$ nomulus -e {ENVIRONMENT} list_reserved_lists
common_bad-words
exampletld_some-other-list
```

## Stackdriver monitoring

[Stackdriver Monitoring](https://cloud.google.com/monitoring/docs/) is used to
instrument internal state within the Nomulus internal environment. This is
broadly called white-box monitoring. Currently, EPP and DNS are instrumented.
The metrics monitored are as follows:

*   `/custom/epp/requests` -- A count of EPP requests, described by command
    name, client id, and return status code.
*   `/custom/epp/processing_time` -- A
    [Distribution](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TypedValue#Distribution)
    representing the processing time for EPP requests, described by command
    name, client id, and retujrn status code.
*   `/custom/dns/publish_domain_requests` -- A count of publish domain requests,
    described by the target TLD and the return status code from the underlying
    DNS implementation.
*   `/custom/dns/publish_host_requests` -- A count of publish host requests,
    described by the target TLD and the return status code from the underlying
    DNS implementation.

Follow the guide to [set up a Stackdriver
account](https://cloud.google.com/monitoring/accounts/guide) and associate it
with the GCP project containing the Nomulus App Engine app. Once the two have
been linked, monitoring will start automatically. For now, because the
visualization of custom metrics in Stackdriver is embryronic, you can retrieve
and visualize the collected metrics with a script, as described in the guide on
[Reading Time
Series](https://cloud.google.com/monitoring/custom-metrics/reading-metrics) and
the [custom metric code
sample](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/monitoring/api/v3/custom_metric.py).

In addition to the included white-box monitoring, black-box monitoring should be
set up to exercise the functionality of the registry platform as a user would
see it. This monitoring should, for example, create a new domain name every few
minutes via EPP and then verify that the domain exists in DNS and WHOIS. For
now, no black-box monitoring implementation is provided with the Nomulus
platform.

## Updating cursors

In most cases, cursors will not advance if a task that utilizes a cursor fails
(so that the task can be retried for that given timestamp). However, there are
some cases where a cursor is updated at the end of a job that produces bad
output (for example, RDE export), and in order to re-run a job, the cursor will
need to be rolled back.

In rare cases it might be useful to roll a cursor forward if there is some bad
data at a given time that prevents a task from completing successfully, and an
acceptable solution is to simply skip the bad data.

Cursors can be updated as follows:

```shell
$ nomulus -e {ENVIRONMENT} update_cursors exampletld --type RDE_STAGING \
    --timestamp 2016-09-01T00:00:00Z
Update Cursor@ahFzfmRvbWFpbi1yZWdpc3RyeXIzCxIPRW50aXR5R3JvdXBSb290Igljcm9zcy10bGQMCxIIUmVnaXN0cnkiB3lvdXR1YmUM_RDE_STAGING
cursorTime -> [2016-09-23T00:00:00.000Z, 2016-09-01T00:00:00.000Z]

Perform this command? (y/N): Y
Updated 1 entities.
```