This removes the following unnecessary imports:
//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
This affects JSR305, JSR330, and Guava annotations.
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
Now that the large zone re-signing test is complete, we no longer need it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
Also increases the number of commit log buckets on alpha to 397 and correspondingly
reduces the frequency of commit log diff exporting to once every 3 minutes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191440586
Datastore has a non-zero chance of failing on reads. A map-reduce with too many
failures will eventually give up. As a result, any map-reduce that goes over a
large number of datastore entities is almost guaranteed to fail.
Since we expect to have a large number of EppResources, we make sure to wrap
all datastore reads with some retrying mechanism to reduce the number of
transient failures that propagate to Map-Reduce.
This feature already existed for CommitLogManifestReader, we refactor the code to use the same retrying mechanism in EppResource readers.
Also removed the transactNew around the reads because looking at the source - it doesn't actually do anything we need (doesn't retry on any failure other than concurrency failure)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190633281
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
Stream.concat only accepts 2 parameters. Streams.concat on the other hand
accepts any number of parameters.
Moving to Streams.concat for all uses (2 or more) makes sense for uniformity
and convenience reasons.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179716648
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
Runnable and Callable are both @FunctionalInterfaces. The difference is
that Callable requires a return value whereas Runnable does not, so in
situations where we don't care about a return value, rather than having to
add an unnecessary 'return null;' at the end of the lambda, we can simply
use a non-returning Runnable instead.
Unfortunately, owing to legacy reasons, Runnable is not declared to throw
checked exceptions whereas Callable is, so in situations where checked
exceptions are thrown we still need to have a 'return null;' call at the
end.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172935400
The infofmt versions only perform string concatenation if info logging
is turned on, rather than doing so all of the time.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172889338
DeleteOldCommitLogs uses the revisions of EPP resources to know which commit logs are still in use.
This means that if the revision list of some EPP resources is corrupt in any way, we might delete commit logs that are still needed.
Here we add a basic sanity check for the revision list: we check it has at least one revision older than the "threshold", where the "threshold" is the oldest date we need to be able to recreate using the commit logs.
If we find an error, we log it (and increment a counter). There's nothing we can to do fix it thought.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172780418
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Lessons from the first full run of the mapreduce.
Fixed the "old manifests still referenced" counter, since some of these
manifests could be new.
Added a debug "manifests incorrectly mapped multiple times" that is expected to never happen, but who knows? (this is a sort of sanity check for the mapreduce)
Added an "Epp resource revisions handled" that counts the total references, allowing for the possibility of the same manifest being referenced from multiple resources - this is expected to be equal to the map's "Epp resource revisions found" counter.
Added a "commit log manifests referenced multiple times" to see how often the same manifest is linked to from different epp resources.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166846879
It appears that there are some possible flows where the reducer runs twice on
the same key. Either because of some error in a subsequent key that makes the
entire shard become ignored and retried, or possible some obscure error outside
of the transaction on that key.
The result however is that sometimes the reducer runs on a key that has already
been deleted. We need to check for that to prevent a null pointer exception.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166112173
Attempting to run DeleteOldCommitLogs in prod resulted in a lot of DatastoreTimeoutException errors. The assumption is that attempting to load so many CommitLogManifests (over 200 million of them), when each one has a slight possibility of failure, has a very high probability of error.
The shard aborts after 20 of these errors, and by eliminating as many loads as possible and retrying the remaining loads inside a transaction we are effectively eliminating any exceptions "leaking" out to the mapreduce framework, which will hopefully keep us bellow 20. At least, that's our best guess currently as to why the mapreduce fails.
EppResources are loaded in the map stage to get the revisions, and CommitLogManifests are only loaded in the reduce stage for sanity check so we don't accidentally delete resources we need in prod. Both of these are wrapped in transactNew to make sure they retry individually.
The only "load" not done inside a transaction is the EppResourceIndex, but there's no getting around that without rewriting the EppResourceInputs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164176764
I know the query that finds commit logs already should ignore commit logs
that are too young, but this adds an explicit sanity check for safety's
sake, so we don't have to depend solely on an indexed query for safety.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162386413
The dry run does all the steps except the deletion. All the counters will
return the same values they would have returned on an actual run.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162379609
This change is motivated by the sandbox run where we saw the backend instances overwhelmed by the 100 default shards to the point where they couldn't even answer a simple status request.
Production has 50 backend instances, so 20 will leave a lot of spare for other tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162357857
We want to be safer and more explicit about the authentication needed by the many actions that exist.
As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
Now instead of deleting "all logs older than X", we delete "all logs older than
X that don't have any EppResource.getRevision()" pointing to them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161455827
Move the "restoreCommitLogs" command from the backend module to the tools
module so it's easier to access with nomulus.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156768389
If there are gaps or forks in the commit log history files, raise an exception
and display the possible sets of files in the log files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155219410
This was an oversight I noticed ages ago, so resurrecting some old local changes I had to correct it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146812322
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072
We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018
A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55
The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.
Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:
- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
This allows separate Bazel projects to reference Nomulus as an external
repository. They can then copy the []
directory structure into their own project and customize the Action
and Module lists for the GAE modules in their own deployment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136863886
This is an internal-only feature that breaks the open source build.
CL created with:
dr-replace '(compatible_with.*)' '\1 # MOE:strip_line'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=128852873
