Attempting to run DeleteOldCommitLogs in prod resulted in a lot of DatastoreTimeoutException errors. The assumption is that attempting to load so many CommitLogManifests (over 200 million of them), when each one has a slight possibility of failure, has a very high probability of error.
The shard aborts after 20 of these errors, and by eliminating as many loads as possible and retrying the remaining loads inside a transaction we are effectively eliminating any exceptions "leaking" out to the mapreduce framework, which will hopefully keep us bellow 20. At least, that's our best guess currently as to why the mapreduce fails.
EppResources are loaded in the map stage to get the revisions, and CommitLogManifests are only loaded in the reduce stage for sanity check so we don't accidentally delete resources we need in prod. Both of these are wrapped in transactNew to make sure they retry individually.
The only "load" not done inside a transaction is the EppResourceIndex, but there's no getting around that without rewriting the EppResourceInputs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164176764
I know the query that finds commit logs already should ignore commit logs
that are too young, but this adds an explicit sanity check for safety's
sake, so we don't have to depend solely on an indexed query for safety.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162386413
The dry run does all the steps except the deletion. All the counters will
return the same values they would have returned on an actual run.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162379609
This change is motivated by the sandbox run where we saw the backend instances overwhelmed by the 100 default shards to the point where they couldn't even answer a simple status request.
Production has 50 backend instances, so 20 will leave a lot of spare for other tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162357857
We want to be safer and more explicit about the authentication needed by the many actions that exist.
As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
Now instead of deleting "all logs older than X", we delete "all logs older than
X that don't have any EppResource.getRevision()" pointing to them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161455827
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
The dark lord Gosling designed the Java package naming system so that
ownership flows from the DNS system. Since we own the domain name
registry.google, it seems only appropriate that we should use
google.registry as our package name.
This change renames directories in preparation for the great package
rename. The repository is now in a broken state because the code
itself hasn't been updated. However this should ensure that git
correctly preserves history for each file.
2016-05-13 18:55:08 -04:00
Renamed from java/com/google/domain/registry/backup/DeleteOldCommitLogsAction.java (Browse further)
