Backup Datastore using the Admin REST API

Add server end points to backup Datastore using managed-export mechanism.
A cron job is defined in Alpha to run daily exports using this implementation.

Existing backup is left running. The new backups are saved to a new set of
locations:
- GCS bucket: gs://PROJECT-ID-datastore-backups
- Big Query data set: datastore_backups
- Big Query latest back up view name: latest_datastore_backup
Also, the names of Bigquery tables now use the export timestamp
assigned by Datastore. E.g., 2018_12_05T23_56_18_50532_ContactResource,

After the new import mechanism is implemented and the back-restore flow is
tested, we will stop the existing backup runs and deploy the new
implementation to all environments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224932957
This commit is contained in:
weiminyu 2018-12-10 20:26:53 -08:00 committed by jianglai
parent ea154a8378
commit 9c706e79fd
27 changed files with 1179 additions and 37 deletions

View file

@ -4,7 +4,9 @@ PATH CLASS METHOD
/_dr/cron/fanout TldFanoutAction GET y INTERNAL APP IGNORED
/_dr/cron/readDnsQueue ReadDnsQueueAction GET y INTERNAL APP IGNORED
/_dr/dnsRefresh RefreshDnsAction GET y INTERNAL APP IGNORED
/_dr/task/backupDatastore BackupDatastoreAction POST y INTERNAL APP IGNORED
/_dr/task/brdaCopy BrdaCopyAction POST y INTERNAL APP IGNORED
/_dr/task/checkDatastoreBackup CheckBackupAction POST,GET y INTERNAL APP IGNORED
/_dr/task/checkSnapshot CheckSnapshotAction POST,GET y INTERNAL APP IGNORED
/_dr/task/copyDetailReports CopyDetailReportsAction POST n INTERNAL,API APP ADMIN
/_dr/task/deleteContactsAndHosts DeleteContactsAndHostsAction GET n INTERNAL APP IGNORED
@ -44,3 +46,4 @@ PATH CLASS METHOD
/_dr/task/tmchDnl TmchDnlAction POST y INTERNAL APP IGNORED
/_dr/task/tmchSmdrl TmchSmdrlAction POST y INTERNAL APP IGNORED
/_dr/task/updateSnapshotView UpdateSnapshotViewAction POST n INTERNAL APP IGNORED
/_dr/task/uploadDatastoreBackup UploadDatastoreBackupAction POST n INTERNAL APP IGNORED