Attempting to run DeleteOldCommitLogs in prod resulted in a lot of DatastoreTimeoutException errors. The assumption is that attempting to load so many CommitLogManifests (over 200 million of them), when each one has a slight possibility of failure, has a very high probability of error.
The shard aborts after 20 of these errors, and by eliminating as many loads as possible and retrying the remaining loads inside a transaction we are effectively eliminating any exceptions "leaking" out to the mapreduce framework, which will hopefully keep us bellow 20. At least, that's our best guess currently as to why the mapreduce fails.
EppResources are loaded in the map stage to get the revisions, and CommitLogManifests are only loaded in the reduce stage for sanity check so we don't accidentally delete resources we need in prod. Both of these are wrapped in transactNew to make sure they retry individually.
The only "load" not done inside a transaction is the EppResourceIndex, but there's no getting around that without rewriting the EppResourceInputs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164176764
Now instead of deleting "all logs older than X", we delete "all logs older than
X that don't have any EppResource.getRevision()" pointing to them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161455827
I'm setting it to three buckets across all tests, because the default one bucket
wasn't realistic enough, and allowed some tests to pass that shouldn't have,
essentially by accident.
This also changes RegistryConfig from being an interface to being an abstract
base class. The medium term goal here is to have it be a static class so that it
can provide fields from the YAML-derived POJO in situations where Dagger
injection isn't feasible.
The expected end state is as follows:
default-config.yaml -- The master config file that provides defaults for all
values.
nomulus-config.yaml -- A per-environment config file that overrides the defaults
from the previous file.
YamlConfig.java -- The POJO that the aforementioned YAML files are deserialized
into.
RegistryConfig.java -- Contains a static, memoized instance of YamlConfig and
provides static methods for getting some of those values.
ConfigModule -- Will become a static inner class of RegistryConfig, using Dagger
to provide most of the fields from the memoized YamlConfig instance. This way,
all configuration will be coming from a single place: RegistryConfig.java.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143567288
The dark lord Gosling designed the Java package naming system so that
ownership flows from the DNS system. Since we own the domain name
registry.google, it seems only appropriate that we should use
google.registry as our package name.
This change renames directories in preparation for the great package
rename. The repository is now in a broken state because the code
itself hasn't been updated. However this should ensure that git
correctly preserves history for each file.
2016-05-13 18:55:08 -04:00
Renamed from javatests/com/google/domain/registry/backup/DeleteOldCommitLogsActionTest.java (Browse further)
