Random used to be a static variable which requires InjectRule to mock it in unit tests. It is now a singleton, which ensures that the same instance is called every time and Random.nextBytes() generates results that distribute uniformly between each call.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217592767
According to RFC 2046, the body of the multipart contains:
multipart-body := [preamble CRLF]
dash-boundary transport-padding CRLF
body-part *encapsulation
close-delimiter transport-padding
[CRLF epilogue]
The preemble and epilogue are optional, and ignored. However, it's not 100%
explicit whether the CRLFs after the preamble and before the epilogue are
required. The one after the preemble is often not given if there's no preemble,
so it's conceivable that you don't *have* to give the CRLF before the epilogue
if there's no epilogue (it's also enclosed in the [], making it part of the
"optional")
However, it seems that when the TMDB "migrated to the cloud" (as they
describe it) on Aug. 13 2018, they started requiring that CRLF.
TESTED=connected to a TMDB-whitelisted server, used CURL to manually create the
message as we currently send it (without the final CRLF) with junk data and got
the error from the bug. Then sent the exact same message with the additional
CRLF, and got a different error that directly relates to the content of the
junk data.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212637246
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).
This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.
This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
This removes the following unnecessary imports:
//third_party/java_src/gtld/java/google/registry/bigquery
//third_party/java_src/gtld/java/google/registry/config
//third_party/java_src/gtld/java/google/registry/cron
//third_party/java_src/gtld/java/google/registry/dns
//third_party/java_src/gtld/java/google/registry/gcs
//third_party/java_src/gtld/java/google/registry/mapreduce
//third_party/java_src/gtld/java/google/registry/model
//third_party/java_src/gtld/java/google/registry/module/backend
//third_party/java_src/gtld/java/google/registry/module/frontend
//third_party/java_src/gtld/java/google/registry/module/pubapi
//third_party/java_src/gtld/java/google/registry/module/tools
//third_party/java_src/gtld/java/google/registry/request
//third_party/java_src/gtld/java/google/registry/security
//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers
//third_party/java_src/gtld/java/google/registry/util
//third_party/java_src/gtld/java/google/registry/xjc
//third_party/java_src/gtld/javatests/google/registry/model
//third_party/java_src/gtld/javatests/google/registry/testing
//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java_src/gtld/java/google/registry/bigquery,//third_party/java_src/gtld/java/google/registry/config,//third_party/java_src/gtld/java/google/registry/cron,//third_party/java_src/gtld/java/google/registry/dns,//third_party/java_src/gtld/java/google/registry/gcs,//third_party/java_src/gtld/java/google/registry/mapreduce,//third_party/java_src/gtld/java/google/registry/model,//third_party/java_src/gtld/java/google/registry/module/backend,//third_party/java_src/gtld/java/google/registry/module/frontend,//third_party/java_src/gtld/java/google/registry/module/pubapi,//third_party/java_src/gtld/java/google/registry/module/tools,//third_party/java_src/gtld/java/google/registry/request,//third_party/java_src/gtld/java/google/registry/security,//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers,//third_party/java_src/gtld/java/google/registry/util,//third_party/java_src/gtld/java/google/registry/xjc,//third_party/java_src/gtld/javatests/google/registry/model,//third_party/java_src/gtld/javatests/google/registry/testing,//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202652421
This removes the following unnecessary imports:
//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
Now that the large zone re-signing test is complete, we no longer need it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
*** Reason for rollback ***
We suspect that this is breaking RDE more, so we're going to rollout a cherrypick of this reversion.
*** Original change description ***
Upload to GCS before uploading to FTP
Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.
In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201005260
Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.
In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199643208
This is a 'yellow' Flogger migration CL. Yellow CLs should be mostly safe
but include changes that are notable for one reason or another. Manual
intervention may be required to address small issues.
The comments in this CL indicate cases where suggested code changes
should be double checked, or even modified. There may even be cases where
files outside this CL are affected by changes to things such as logger
visibility. However if a change does not have an associated comment then
it should be safe.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097990
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
'afterFinalFailure' is called just before rethrowing a non-retrying error from
the retrier. This can happen either because the exception shouldn't be retried,
or because we exceeded the maximum number of retries.
The same thing can be done by catching that thrown error outside of the
retrier:
retrier.callWithRetry(
callable,
new FailureReporter() {
@Override
void afterFinalFailure(Throwable thrown, int failures) {
// do something with thrown
}
},
RetriableException.class);
is (almost) the same as:
try {
retrier.callWithRetry(callable, RetriableException.class);
} catch (Throwable thrown) {
// do something with thrown
throw thrown;
}
("almost" because the retrier might wrap the Throwable in a RuntimeException,
so you might need to getCause or getRootCause. Also - there is the
"beforeRetry" I ignored for the example)
Removing "afterFinalFailure" also makes the FailureReporter in line with Java 8
functional interface - meaning we can more easily create it when we do need to
override "beforeRetry".
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189972101
This enables sharded DNS publishing on a per-TLD basis. Instead of a TLD-wide lock, the sharded scheme locks each update on the shard number, allowing parallel writes to DNS.
We allow N (the number of shards) to be 0 or 1 for no sharding, and N > 1 for an N-way sharding scheme. Unless explicitly set, all TLDs default to a numShards of 0, so we don't have to reload all registry objects explicitly.
WARNING: This will change the lock name upon deployment for the PublishDnsAction from "<TLD> Dns Updates" to "<TLD> Dns Updates shard 0". This may cause concurrency issues if the underlying DNSWriter is not parallel-write tolerant (currently all production usages are ZonemanWriter, which is parallel-tolerant, so no issues are expected).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187525655
This is in preparation for running the automatic refactoring script that
will replace all ExpectedExceptions with use of JUnit 4.13's assertThrows/
expectThrows.
Note that I have recorded the callsites of assertions about EppExceptions
being marshallable and will edit those specific assertions back in after
running the automatic refactoring script (which do not understand these).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178812403
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.
Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Tools inheriting from MutatingCommand print out the change they are going to
make and then ask the user to confirm that this is indeed what they wanted to
do.
The change is outputted as a list of updated values in the form
key -> [oldValue, newValue]
e.g.
dnsPaused -> [true, false]
This CL will change the output to be clearer:
key: oldValue -> newValue
e.g.
dnsPaused: true -> false
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170853745
When trying to run the MapReduce for DeleteOldCommitLogsAction, we run into a
lot of DatastoreTimeoutException during CommitLogManifestReader.next.
This causes the entire shard to fail. Since we have a lot of keys (tens of
millions), this is almost guaranteed to happen, dooming the entire MapReduce.
Here is an attempt to recover from the Timeout Exception by saving the state
before the read, then on failure restoring that state and trying again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165172222
This standardizes use of annotations/inheritance/formatting across
tests, to make the code more legible and consistent.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161810734
hasMessageThat() was added in Truth 0.32 and we are already on that
version in the Nomulus release.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153095111
Replace KeystoreKeyring with ComparatorKeyring between KeystoreKeyring and
KmsKeystore. In the opensource version, will replace DummyKeyring with
KmsKeyring directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152893767
There will now be "resultToString" and "resultEquals" to stringify / compare
the method results.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150135475
This is an alternative to the "manually constructed" version in []
This generator recieves two objects implementing some interface. It returns a new implementation that exactly duplicates the behavior (return values, exceptions etc.) of the first ("original") object, but also logs any differences (in either return value or thrown exception) from the second object.
Nothing in the second object's execution will affect the results - not even thrown exceptions - it's only used for logging any differences.
This is useful when switching backend architecture for the some interface (say... from keystore to datastore+KMS for the Keyring interface :)) as we can make sure everything was OK for some extended period of real data before doing the switch.
This is better than just comparing the outputs once because having a completely different backend out of our control can result in "glitches" that are rare but relevant: maybe the new backend has lower availability, or sometimes fails and gives wrong results, or... well, that's it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149706165
This allows us to use util methods from within config, which is a useful thing
to be able to do for, e.g., being able to log errors while loading configuration.
It makes sense that the util package should be at the very base of the
class inheritance hierarchy; config seems logically higher than it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=144324273
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
This primarily addresses issues with TMCH testing mode and email sending utils.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143710550
We're only using it for generating multiparty boundaries, and there's no real need for the random boundary values to be cryptographically secure. The point of the randomness is just to make collisions with content in the payload sufficiently unlikely. The app itself controls the payload contents, and while it might be derived from user-submitted content, in practice it would be nearly infeasible to get the payload to contain arbitrary boundary values even if the RNG-produced boundaries could be determined in advance.
To further insulate against this, I've increased the boundary size (from 40 bits to 192) and added an actual check that the boundary isn't present in the input data, so that in the extremely unlikely event of a collision, we fail rather than producing an invalid multipart request.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142784289
