Random used to be a static variable which requires InjectRule to mock it in unit tests. It is now a singleton, which ensures that the same instance is called every time and Random.nextBytes() generates results that distribute uniformly between each call.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217592767
This allows us to inject it with Dagger and avoid using InjectRule to set it
in unit tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217571475
According to RFC 2046, the body of the multipart contains:
multipart-body := [preamble CRLF]
dash-boundary transport-padding CRLF
body-part *encapsulation
close-delimiter transport-padding
[CRLF epilogue]
The preemble and epilogue are optional, and ignored. However, it's not 100%
explicit whether the CRLFs after the preamble and before the epilogue are
required. The one after the preemble is often not given if there's no preemble,
so it's conceivable that you don't *have* to give the CRLF before the epilogue
if there's no epilogue (it's also enclosed in the [], making it part of the
"optional")
However, it seems that when the TMDB "migrated to the cloud" (as they
describe it) on Aug. 13 2018, they started requiring that CRLF.
TESTED=connected to a TMDB-whitelisted server, used CURL to manually create the
message as we currently send it (without the final CRLF) with junk data and got
the error from the bug. Then sent the exact same message with the additional
CRLF, and got a different error that directly relates to the content of the
junk data.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212637246
We never used it and don't have any plans to use it going forward. All
conceivable parts of its functionality that we might use going forward have
already been subsumed into allocation tokens, which are a simpler way of
handling the same use case that are also standards-compliant.
Also gets rid of the hideous ANCHOR_ prefix on anchor tenant EPP authcodes
that was only ever necessary because of overloading the authcode for
anchor tenant creation. Going forward it'll be based on allocation tokens,
so there's no risk of conflicts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209418194
The "tar file encoding" saves the file + metadata (filename and modification) in a "tar" format that is required in the RDE spec, even though it only contains a single file.
This is only relevant for RyDE, and not for Ghostryde. In fact, the only reason Ghostryde exists is to not have the TAR layer.
Currently we only encrypt RyDE, so we only need the TAR encoding. We plan to add decryption ability so we can test files we sent to IronMountain if there's a problem - so we will need TAR decoding for that.
The new file - RydeTar.java - has both encoding and decoding. We keep the format used for all other Input/OutputStreams for consistency, even though in this case it could be a private part of the RyDE encoder / decoder.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208056757
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).
This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.
This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
This removes the following unnecessary imports:
//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
This affects JSR305, JSR330, and Guava annotations.
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
We'll continue to use injected clocks for the rest of our tests that use signed marks files, so that they don't all fail after the current validity period. The new test TmchTestDataExpirationTest will let us know when the files are expired, so we can update them.
All updated test data files come from https://newgtlds.icann.org/en/about/trademark-clearinghouse/registries-registrars
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202208196
Two main changes:
- Replaced getClass().getSimpleName() in the logs with a constructor-given name. Right now what we have is a lot of identical classes with slightly different names so that the logs would be different. With this change - we can later get rid of a lot of these classes and replace them with simple wrappers.
- Removed the "expected" feature. Only Tar uses it - and it can override onClose to do that (that's what it's there for!)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202129563
It doesn't entirely make semantic sense, since the actual state of the
SystemClock isn't being preserved, but it makes injection into serializable
classes (e.g. mapreduces) much simpler, so it's worth doing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201755949
Now that the large zone re-signing test is complete, we no longer need it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
*** Reason for rollback ***
We suspect that this is breaking RDE more, so we're going to rollout a cherrypick of this reversion.
*** Original change description ***
Upload to GCS before uploading to FTP
Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.
In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201005260
Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.
In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199643208
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
This is a 'yellow' Flogger migration CL. Yellow CLs should be mostly safe
but include changes that are notable for one reason or another. Manual
intervention may be required to address small issues.
The comments in this CL indicate cases where suggested code changes
should be double checked, or even modified. There may even be cases where
files outside this CL are affected by changes to things such as logger
visibility. However if a change does not have an associated comment then
it should be safe.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097990
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
Locales are weird. Even if all our character individually are just 0-9a-z_,
different locales might still convert them differently to upper/lower cases...
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193512312
'afterFinalFailure' is called just before rethrowing a non-retrying error from
the retrier. This can happen either because the exception shouldn't be retried,
or because we exceeded the maximum number of retries.
The same thing can be done by catching that thrown error outside of the
retrier:
retrier.callWithRetry(
callable,
new FailureReporter() {
@Override
void afterFinalFailure(Throwable thrown, int failures) {
// do something with thrown
}
},
RetriableException.class);
is (almost) the same as:
try {
retrier.callWithRetry(callable, RetriableException.class);
} catch (Throwable thrown) {
// do something with thrown
throw thrown;
}
("almost" because the retrier might wrap the Throwable in a RuntimeException,
so you might need to getCause or getRootCause. Also - there is the
"beforeRetry" I ignored for the example)
Removing "afterFinalFailure" also makes the FailureReporter in line with Java 8
functional interface - meaning we can more easily create it when we do need to
override "beforeRetry".
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189972101
This enables sharded DNS publishing on a per-TLD basis. Instead of a TLD-wide lock, the sharded scheme locks each update on the shard number, allowing parallel writes to DNS.
We allow N (the number of shards) to be 0 or 1 for no sharding, and N > 1 for an N-way sharding scheme. Unless explicitly set, all TLDs default to a numShards of 0, so we don't have to reload all registry objects explicitly.
WARNING: This will change the lock name upon deployment for the PublishDnsAction from "<TLD> Dns Updates" to "<TLD> Dns Updates shard 0". This may cause concurrency issues if the underlying DNSWriter is not parallel-write tolerant (currently all production usages are ZonemanWriter, which is parallel-tolerant, so no issues are expected).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187525655
To make FOSS build compile, third_party vendoring rules for jaxb are added to package all jaxb related targets imported from maven into a uber jar, mirroring the same practice done in //third_party/java/jaxb
Cloned from CL 182666460 by 'g4 patch'.
Original change by cushon@cushon:rosie182283995-0071_Rosie:47348:citc on 2018/01/20 13:36:15.
More information:
https://docs.google.com/document/d/1htErgDIoHMEuMBfGwrtS_O4WwhTw8QOGLva-7aYYvYs/edit?usp=sharing
Tested:
TAP --sample for global presubmit queue
[] passed FOSS test
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182855173
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
This moves the new pipeline's invoice generation to the billing bucket, under the 'invoices/yyyy-MM' subdirectory.
This also changes the invoice e-mail to use a multipart message that attaches the invoice to the e-mail, to guarantee the correct MIME type and download.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181746191
This creates a specified number of tokens of a given schema, with a dryrun option
to not persist them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181403775
Before pushing an update to Cloud DNS, the CloudDnsWriter needs to read all the domain RRSs from Cloud DNS one by one to know what to delete.
Doing so sequentially results in update times that are too long (approx 200ms per domain, which is 20 seconds per batch of 100) severely limiting our QPS.
This CL uses Concurrent threading to do the Cloud DNS queries in parallel. Unfortunately, my preferred method (Set.parallelStream) doesn't work on App Engine :(
This reduces the per-item time from 200ms to 80ms, which can be further reduced to 50ms if we remove the rate limiter (currently set to 20 per second).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178126877
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
Runnable and Callable are both @FunctionalInterfaces. The difference is
that Callable requires a return value whereas Runnable does not, so in
situations where we don't care about a return value, rather than having to
add an unnecessary 'return null;' at the end of the lambda, we can simply
use a non-returning Runnable instead.
Unfortunately, owing to legacy reasons, Runnable is not declared to throw
checked exceptions whereas Callable is, so in situations where checked
exceptions are thrown we still need to have a 'return null;' call at the
end.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172935400
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Tools inheriting from MutatingCommand print out the change they are going to
make and then ask the user to confirm that this is indeed what they wanted to
do.
The change is outputted as a list of updated values in the form
key -> [oldValue, newValue]
e.g.
dnsPaused -> [true, false]
This CL will change the output to be clearer:
key: oldValue -> newValue
e.g.
dnsPaused: true -> false
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170853745
We're going to need to switch away from Guava's Functions and Predicates for
everything and replace them with the java.util versions. Unfortunately there
does not appear to be an automated tool to do this all at once. Refaster got
close but doesn't seem to care about these particular types of mismatch (I
suspect we're using a different version of the JDK than the outside world;
ours is OK with Guava classes).
This also bumps up Guava to 0.23, which is needed for some new functionality
used in combination with Java 8 features.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170531539
This is needed for the Lock.java enhancement where a lock will be implicitly
released if the request owning it dies.
No matter which solution we want for refactoring the Lock itself, we will need this class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167600314
When trying to run the MapReduce for DeleteOldCommitLogsAction, we run into a
lot of DatastoreTimeoutException during CommitLogManifestReader.next.
This causes the entire shard to fail. Since we have a lot of keys (tens of
millions), this is almost guaranteed to happen, dooming the entire MapReduce.
Here is an attempt to recover from the Timeout Exception by saving the state
before the read, then on failure restoring that state and trying again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165172222
This will make DNS issues easier to debug retroactively as we will be
able to determine, by looking at the logs, if the queue size was growing
unbounded.
Also adds some logging helpers to allow programmatically choosing the level
of logging.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163123783
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).
This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
We currently only have one method for fine level logging. Add another three so that fine level logging can be called just like other levels.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158760803
