Add Cursor documentation

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133994188
This commit is contained in:
ctingue 2016-09-22 13:42:36 -07:00 committed by Ben McIlwain
parent b783acfcc6
commit 414781e79b
2 changed files with 61 additions and 0 deletions

View file

@ -13,6 +13,43 @@ particularly important pieces of the system are implemented.
## Cursors
Cursors are `DateTime` pointers used to ensure rolling transactional isolation
of various reporting and other maintenance operations. Utilizing a `Cursor`
within an operation ensures that instances in time are processed exactly once
for a given task, and that tasks can catch up from any failure states at any
time.
Cursors are rolled forward at the end of successful tasks, are not rolled
forward in the case of failure, and can be manually set backwards using the
update_cursors command in registry_tool to reprocess a past action.
The following cursor types are defined:
* **`BRDA`** - BRDA (thin) escrow deposits
* **`RDE_REPORT`** - XML RDE report uploads
* **`RDE_STAGING`** - RDE (thick) escrow deposit staging
* **`RDE_UPLOAD`** - RDE (thick) escrow deposit upload
* **`RDE_UPLOAD_SFTP`** - Cursor that tracks the last time we talked to the
escrow provider's SFTP server for a given TLD.
* **`RECURRING_BILLING`** - Expansion of `Recurring` (renew) billing events
into `OneTime` events.
All `Cursor` entities in Datastore contain a `DateTime` that represents the next
timestamp at which an operation should resume processing and a `CursorType` that
identifies which operation the cursor is associated with. In many cases, there
are multiple cursors per operation; for instance, the cursors related to RDE
reporting, staging, and upload are per-TLD cursors. To accomplish this, each
`Cursor` also has a scope, a `Key<ImmutableObject>` to which the particular
cursor applies (this can be e.g. a `Registry` or any other `ImmutableObject` in
datastore, depending on the operation). If the `Cursor` applies to the entire
registry environment, it is considered a global cursor and has a scope of
`EntityGroupRoot.getCrossTldKey()`.
Cursors are singleton entities by type and scope. The id for a `Cursor` is a
deterministic string that consists of the websafe string of the Key of the scope
object concatenated with the name of the name of the cursor type, separated by
an underscore.
## Mapreduces
## Actions and servlets

View file

@ -266,3 +266,27 @@ exampletld_some-other-list
```
## StackDriver monitoring
## Updating cursors
In most cases, cursors will not advance if a task that utilizes a cursor fails
(so that the task can be retried for that given timestamp). However, there are
some cases where a cursor is updated at the end of a job that produces bad
output (for example, RDE export), and in order to re-run a job, the cursor will
need to be rolled back.
In rare cases it might be useful to roll a cursor forward if there is some bad
data at a given time that prevents a task from completing successfully, and an
acceptable solution is to simply skip the bad data.
Cursors can be updated as follows:
```shell
$ registry_tool -e {ENVIRONMENT} update_cursors exampletld --type RDE_STAGING \
--timestamp 2016-09-01T00:00:00Z
Update Cursor@ahFzfmRvbWFpbi1yZWdpc3RyeXIzCxIPRW50aXR5R3JvdXBSb290Igljcm9zcy10bGQMCxIIUmVnaXN0cnkiB3lvdXR1YmUM_RDE_STAGING
cursorTime -> [2016-09-23T00:00:00.000Z, 2016-09-01T00:00:00.000Z]
Perform this command? (y/N): Y
Updated 1 entities.
```