From 414781e79bd556770d9c6a4fd499ef20a68d55ee Mon Sep 17 00:00:00 2001
From: ctingue <ctingue@google.com>
Date: Thu, 22 Sep 2016 13:42:36 -0700
Subject: [PATCH] Add Cursor documentation

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133994188
---
 docs/code-structure.md         | 37 ++++++++++++++++++++++++++++++++++
 docs/operational-procedures.md | 24 ++++++++++++++++++++++
 2 files changed, 61 insertions(+)

diff --git a/docs/code-structure.md b/docs/code-structure.md
index 6de055532..83bd64347 100644
--- a/docs/code-structure.md
+++ b/docs/code-structure.md
@@ -13,6 +13,43 @@ particularly important pieces of the system are implemented.
 
 ## Cursors
 
+Cursors are `DateTime` pointers used to ensure rolling transactional isolation
+of various reporting and other maintenance operations. Utilizing a `Cursor`
+within an operation ensures that instances in time are processed exactly once
+for a given task, and that tasks can catch up from any failure states at any
+time.
+
+Cursors are rolled forward at the end of successful tasks, are not rolled
+forward in the case of failure, and can be manually set backwards using the
+update_cursors command in registry_tool to reprocess a past action.
+
+The following cursor types are defined:
+
+*   **`BRDA`** - BRDA (thin) escrow deposits
+*   **`RDE_REPORT`** - XML RDE report uploads
+*   **`RDE_STAGING`** - RDE (thick) escrow deposit staging
+*   **`RDE_UPLOAD`** - RDE (thick) escrow deposit upload
+*   **`RDE_UPLOAD_SFTP`** - Cursor that tracks the last time we talked to the
+    escrow provider's SFTP server for a given TLD.
+*   **`RECURRING_BILLING`** - Expansion of `Recurring` (renew) billing events
+    into `OneTime` events.
+
+All `Cursor` entities in Datastore contain a `DateTime` that represents the next
+timestamp at which an operation should resume processing and a `CursorType` that
+identifies which operation the cursor is associated with. In many cases, there
+are multiple cursors per operation; for instance, the cursors related to RDE
+reporting, staging, and upload are per-TLD cursors. To accomplish this, each
+`Cursor` also has a scope, a `Key<ImmutableObject>` to which the particular
+cursor applies (this can be e.g. a `Registry` or any other `ImmutableObject` in
+datastore, depending on the operation). If the `Cursor` applies to the entire
+registry environment, it is considered a global cursor and has a scope of
+`EntityGroupRoot.getCrossTldKey()`.
+
+Cursors are singleton entities by type and scope. The id for a `Cursor` is a
+deterministic string that consists of the websafe string of the Key of the scope
+object concatenated with the name of the name of the cursor type, separated by
+an underscore.
+
 ## Mapreduces
 
 ## Actions and servlets
diff --git a/docs/operational-procedures.md b/docs/operational-procedures.md
index e819d7898..8b8e28a5e 100644
--- a/docs/operational-procedures.md
+++ b/docs/operational-procedures.md
@@ -266,3 +266,27 @@ exampletld_some-other-list
 ```
 
 ## StackDriver monitoring
+
+## Updating cursors
+
+In most cases, cursors will not advance if a task that utilizes a cursor fails
+(so that the task can be retried for that given timestamp). However, there are
+some cases where a cursor is updated at the end of a job that produces bad
+output (for example, RDE export), and in order to re-run a job, the cursor will
+need to be rolled back.
+
+In rare cases it might be useful to roll a cursor forward if there is some bad
+data at a given time that prevents a task from completing successfully, and an
+acceptable solution is to simply skip the bad data.
+
+Cursors can be updated as follows:
+
+```shell
+$ registry_tool -e {ENVIRONMENT} update_cursors exampletld --type RDE_STAGING \
+    --timestamp 2016-09-01T00:00:00Z
+Update Cursor@ahFzfmRvbWFpbi1yZWdpc3RyeXIzCxIPRW50aXR5R3JvdXBSb290Igljcm9zcy10bGQMCxIIUmVnaXN0cnkiB3lvdXR1YmUM_RDE_STAGING
+cursorTime -> [2016-09-23T00:00:00.000Z, 2016-09-01T00:00:00.000Z]
+
+Perform this command? (y/N): Y
+Updated 1 entities.
+```