Upload to GCS before uploading to FTP

Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.

In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199643208
This commit is contained in:
guyben 2018-06-07 09:03:52 -07:00 committed by Ben McIlwain
parent 228e4f6c95
commit ea08661598
5 changed files with 43 additions and 181 deletions

View file

@ -87,14 +87,15 @@ class EscrowTaskRunner {
final Duration interval) {
Callable<Void> lockRunner =
() -> {
logger.atInfo().log("TLD: %s", registry.getTld());
DateTime startOfToday = clock.nowUtc().withTimeAtStartOfDay();
Cursor cursor = ofy().load().key(Cursor.createKey(cursorType, registry)).now();
logger.atInfo().log(
"TLD: %s, cursorType: %s cursor: %s", registry.getTld(), cursorType, cursor);
final DateTime nextRequiredRun = (cursor == null ? startOfToday : cursor.getCursorTime());
if (nextRequiredRun.isAfter(startOfToday)) {
throw new NoContentException("Already completed");
}
logger.atInfo().log("Cursor: %s", nextRequiredRun);
logger.atInfo().log("CursorTime: %s", nextRequiredRun);
task.runWithLock(nextRequiredRun);
ofy()
.transact(