Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.
In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199643208
This is a 'yellow' Flogger migration CL. Yellow CLs should be mostly safe
but include changes that are notable for one reason or another. Manual
intervention may be required to address small issues.
The comments in this CL indicate cases where suggested code changes
should be double checked, or even modified. There may even be cases where
files outside this CL are affected by changes to things such as logger
visibility. However if a change does not have an associated comment then
it should be safe.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097990
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
'afterFinalFailure' is called just before rethrowing a non-retrying error from
the retrier. This can happen either because the exception shouldn't be retried,
or because we exceeded the maximum number of retries.
The same thing can be done by catching that thrown error outside of the
retrier:
retrier.callWithRetry(
callable,
new FailureReporter() {
@Override
void afterFinalFailure(Throwable thrown, int failures) {
// do something with thrown
}
},
RetriableException.class);
is (almost) the same as:
try {
retrier.callWithRetry(callable, RetriableException.class);
} catch (Throwable thrown) {
// do something with thrown
throw thrown;
}
("almost" because the retrier might wrap the Throwable in a RuntimeException,
so you might need to getCause or getRootCause. Also - there is the
"beforeRetry" I ignored for the example)
Removing "afterFinalFailure" also makes the FailureReporter in line with Java 8
functional interface - meaning we can more easily create it when we do need to
override "beforeRetry".
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189972101
This enables sharded DNS publishing on a per-TLD basis. Instead of a TLD-wide lock, the sharded scheme locks each update on the shard number, allowing parallel writes to DNS.
We allow N (the number of shards) to be 0 or 1 for no sharding, and N > 1 for an N-way sharding scheme. Unless explicitly set, all TLDs default to a numShards of 0, so we don't have to reload all registry objects explicitly.
WARNING: This will change the lock name upon deployment for the PublishDnsAction from "<TLD> Dns Updates" to "<TLD> Dns Updates shard 0". This may cause concurrency issues if the underlying DNSWriter is not parallel-write tolerant (currently all production usages are ZonemanWriter, which is parallel-tolerant, so no issues are expected).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187525655
This is in preparation for running the automatic refactoring script that
will replace all ExpectedExceptions with use of JUnit 4.13's assertThrows/
expectThrows.
Note that I have recorded the callsites of assertions about EppExceptions
being marshallable and will edit those specific assertions back in after
running the automatic refactoring script (which do not understand these).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178812403
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.
Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Tools inheriting from MutatingCommand print out the change they are going to
make and then ask the user to confirm that this is indeed what they wanted to
do.
The change is outputted as a list of updated values in the form
key -> [oldValue, newValue]
e.g.
dnsPaused -> [true, false]
This CL will change the output to be clearer:
key: oldValue -> newValue
e.g.
dnsPaused: true -> false
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170853745
When trying to run the MapReduce for DeleteOldCommitLogsAction, we run into a
lot of DatastoreTimeoutException during CommitLogManifestReader.next.
This causes the entire shard to fail. Since we have a lot of keys (tens of
millions), this is almost guaranteed to happen, dooming the entire MapReduce.
Here is an attempt to recover from the Timeout Exception by saving the state
before the read, then on failure restoring that state and trying again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165172222
This standardizes use of annotations/inheritance/formatting across
tests, to make the code more legible and consistent.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161810734
hasMessageThat() was added in Truth 0.32 and we are already on that
version in the Nomulus release.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153095111
Replace KeystoreKeyring with ComparatorKeyring between KeystoreKeyring and
KmsKeystore. In the opensource version, will replace DummyKeyring with
KmsKeyring directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152893767
There will now be "resultToString" and "resultEquals" to stringify / compare
the method results.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150135475
This is an alternative to the "manually constructed" version in []
This generator recieves two objects implementing some interface. It returns a new implementation that exactly duplicates the behavior (return values, exceptions etc.) of the first ("original") object, but also logs any differences (in either return value or thrown exception) from the second object.
Nothing in the second object's execution will affect the results - not even thrown exceptions - it's only used for logging any differences.
This is useful when switching backend architecture for the some interface (say... from keystore to datastore+KMS for the Keyring interface :)) as we can make sure everything was OK for some extended period of real data before doing the switch.
This is better than just comparing the outputs once because having a completely different backend out of our control can result in "glitches" that are rare but relevant: maybe the new backend has lower availability, or sometimes fails and gives wrong results, or... well, that's it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149706165
This allows us to use util methods from within config, which is a useful thing
to be able to do for, e.g., being able to log errors while loading configuration.
It makes sense that the util package should be at the very base of the
class inheritance hierarchy; config seems logically higher than it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=144324273
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
This primarily addresses issues with TMCH testing mode and email sending utils.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143710550
We're only using it for generating multiparty boundaries, and there's no real need for the random boundary values to be cryptographically secure. The point of the randomness is just to make collisions with content in the payload sufficiently unlikely. The app itself controls the payload contents, and while it might be derived from user-submitted content, in practice it would be nearly infeasible to get the payload to contain arbitrary boundary values even if the RNG-produced boundaries could be determined in advance.
To further insulate against this, I've increased the boundary size (from 40 bits to 192) and added an actual check that the boundary isn't present in the input data, so that in the extremely unlikely event of a collision, we fail rather than producing an invalid multipart request.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142784289
To add additional logic for flow code, write custom classes that extend the existing custom logic classes (of which DomainCreateFlowCustomLogic is the first provided example), along with a class that extends CustomLogicFactory to provide instances of the new custom logic classes. Then configure the fully qualified class name of your new custom logic factory in ConfigModule.provideCustomLogicFactoryClass().
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=139221577
This removes the countless lines of the form "[null, []]" in registry_tool diffs
that are an artifact of the way we handle nulls in Objectify.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133409440
Java's stock regex implementation doesn't guarantee linear time
complexity which makes it a security liability.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=121159875
The dark lord Gosling designed the Java package naming system so that
ownership flows from the DNS system. Since we own the domain name
registry.google, it seems only appropriate that we should use
google.registry as our package name.
This change renames directories in preparation for the great package
rename. The repository is now in a broken state because the code
itself hasn't been updated. However this should ensure that git
correctly preserves history for each file.
