Make every dependency request explicit on what encoding is used. Also get rid of InjectRule in XjcToDomainResourceConverterTest.
Random number generator providers are separated to secure and insecure ones. The insecure ones must be explicitly requested (usually for use cases where security is not of concern, for better speed).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217921422
The "tar file encoding" saves the file + metadata (filename and modification) in a "tar" format that is required in the RDE spec, even though it only contains a single file.
This is only relevant for RyDE, and not for Ghostryde. In fact, the only reason Ghostryde exists is to not have the TAR layer.
Currently we only encrypt RyDE, so we only need the TAR encoding. We plan to add decryption ability so we can test files we sent to IronMountain if there's a problem - so we will need TAR decoding for that.
The new file - RydeTar.java - has both encoding and decoding. We keep the format used for all other Input/OutputStreams for consistency, even though in this case it could be a private part of the RyDE encoder / decoder.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208056757
The "file encoding" saves the file + metadata (filename and modification) in a "blob" format that PGP knows how to read.
Merges the file-encoder creation between RyDE and Ghostryde.
The new file - RydeFileEncoding.java - is a merge of the removed functions in
Ghostryde.java and the RydePgpFileOutputStream.java.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205295756
Merges the encryptor creation between RyDE and Ghostryde.
The new file - RydeEncryption.java - is a merge of the removed functions in
Ghostryde.java and the RydePgpEncryptionOutputStream.java.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205246053
Merges the compressor creation between RyDE and Ghostryde. Note that GhostRyde
will now compress with ZIP rather than the previous ZLIB. This is backwards
compatible because the decompression algorithm works with either, so files
created by the old version (with ZLIB) can still be opened by the new version,
and vice-versa.
The new file - RydeCompression.java - is a merge of the removed functions in Ghostryde.java and the RydePgpCompressionOutputStream.java.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205102150
Ghostryde.java has a lot of duplicate code with RydeEncoder and the future
RydeDecoder - the encryption/decryption, compression/decompression, file
encoding/decoding. The "de-XXX" part of each of these pairs needs to read a PGP
object from a stream using PGPObjectFactory.
Since we want to move the duplicate code into their own files, we will need to
move the "read PGP objects from stream" functions to a common utility class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205092800
Second step of RDE encoding refactoring.
Creates a single OutputStream encode RyDE files.
This replaces the 5 OutputStreams that were needed before.
Also removes all the factories that were injected. It's an encoding, there's no point in injecting it.
Finally, removed the buffer-size configuration and replaced with a static final
const value in each individual OutputStream.
This doesn't yet include a decoder (InputStream). And there's still a lot of overlap between the Ryde and the Ghostryde code. Both of those are left for the next CLs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204898369
First step of RDE encoding refactoring.
Creates a single InputStream (OutputStream) to decode (encode) Ghostryde files.
This replaces the 3 InputStreams (OutputStreams) that were needed before.
Also removes a lot of classes, and removes the "injection" of the Ghostryde
class. It's an encoding, there's no point in injecting it.
Finally, removed the buffer-size configuration and replaced with a static final
const value. It's just a buffer size - it doesn't actually affect much. There
are much more "important" fields that weren't configured (such as the
compression algorithm and whether or not to do integrity checks)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202319102
Two main changes:
- Replaced getClass().getSimpleName() in the logs with a constructor-given name. Right now what we have is a lot of identical classes with slightly different names so that the logs would be different. With this change - we can later get rid of a lot of these classes and replace them with simple wrappers.
- Removed the "expected" feature. Only Tar uses it - and it can override onClose to do that (that's what it's there for!)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202129563
This is one last hanging piece of work left over from last year's Java 8
migration. There's no functionality changes in this CL, just refactoring.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201947600
New upload tasks are created every 4 hours, so if we're waiting on a 2 hour SFTP cooldown or some other long-running dependency like generating the RDE report, just delete this task and let it re-run at the next 4 hour period. No need to let these tasks continue gumming up the queue.
Note that this method of throwing NoContentException to abort the task without enqueuing it for retry is already being used by RdeReportAction for the same purpose.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201372808
Now that the large zone re-signing test is complete, we no longer need it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
*** Reason for rollback ***
We suspect that this is breaking RDE more, so we're going to rollout a cherrypick of this reversion.
*** Original change description ***
Upload to GCS before uploading to FTP
Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.
In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201005260
Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.
In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199643208
Currently, we have two different ways to parse a "set" parameter:
key=value1&key=value2&key=value3...
and
keys=value1,value2,value3
This is error prone for several reasons:
- different parts of the code must be "synchronized" to use the same style (the
place that creates the request, and the place that parses the request)
- for the key=value1&key=value2, we often use the same key name for the single
value and the set value. This can result in subtle bugs where part of the
code will successfully read the key assuming there's only one key (and will
get the first key=value1, ignoring the rest)
Here we transition everything to the keys=value1,value2,value3 method. This one
was chosen because:
- it's shorter
- it's more intuitive for users
- the key name is plural, differentiating it from the singular key=value that
other requests might need
-----------------------------------
To make sure there are not "transition issues", we will continue to support
(with warnings) the key=value1&key=value2 parameter parsing until we're sure we
haven't forgotten to update any part of the code.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198810681
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
This is a 'red' Flogger migration CL. Red CLs contain changes which are
likely not to work without manual intervention.
Note that it may not even be possible to directly migrate the logger
usage in this CL to the Flogger API and some additional refactoring may
be required. If this is the case, please note that it should be safe to
submit any outstanding 'green' and 'yellow' CLs prior to tackling this.
If you feel that your use case is not covered by the existing Flogger API
please raise a feature request at []and
revert this CL.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198463651
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
To make FOSS build compile, third_party vendoring rules for jaxb are added to package all jaxb related targets imported from maven into a uber jar, mirroring the same practice done in //third_party/java/jaxb
Cloned from CL 182666460 by 'g4 patch'.
Original change by cushon@cushon:rosie182283995-0071_Rosie:47348:citc on 2018/01/20 13:36:15.
More information:
https://docs.google.com/document/d/1htErgDIoHMEuMBfGwrtS_O4WwhTw8QOGLva-7aYYvYs/edit?usp=sharing
Tested:
TAP --sample for global presubmit queue
[] passed FOSS test
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182855173
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
Remove the hack that allows us to use JSch with Java 7 on App Engine -
basically, we have a modified version of JSch that lets us attach a
GAE-friendly thread factory and use that for all JSch threads.
TESTED: Verified that RDE SFTP uploads still work on alpha.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182208225
This creates a specified number of tokens of a given schema, with a dryrun option
to not persist them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181403775
This is needed to fix an inability in Java 8 to correctly infer the type
when the transaction was being allowed to return the value it loaded. The
error was:
INFO: Compilation unit has error diagnostics: [third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:109: error: incompatible types: inference variable R has incompatible bounds
ofy().transact(() -> rdeImportUtils.importEppResource(newContact));
^
upper bounds: java.lang.Object
lower bounds: void, third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:132:
error: incompatible types: inference variable R has incompatible bounds
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179082154
It was easier to simply move these over manually than to try to debug
the automated tooling.
I also changed the case in an exception message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178926365
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
This also improves performance by calculating a set once rather than
on-demand every time it is used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177471572
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
Runnable and Callable are both @FunctionalInterfaces. The difference is
that Callable requires a return value whereas Runnable does not, so in
situations where we don't care about a return value, rather than having to
add an unnecessary 'return null;' at the end of the lambda, we can simply
use a non-returning Runnable instead.
Unfortunately, owing to legacy reasons, Runnable is not declared to throw
checked exceptions whereas Callable is, so in situations where checked
exceptions are thrown we still need to have a 'return null;' call at the
end.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172935400
The TLD is added separately to the lock (the lock is unique per
TLD-ResourceName pair), so there's no need to add it to the resourceName.
The current status is that the TLD was included twice in the lockId. After the fix - it'll only be included once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172659986
Unfortunately this tool isn't smart enough to deal with the assertThat
situation (which has two static imports of a function with the same name).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172000753
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
This CL adds transferredRegistrationExpirationTime as a TransferData field
persisted to Datastore. It's only relevant for domains, and it represents the
registration expiration time resulting from the approval of the most recent
transfer request. For pending transfers, we assume the transfer will be
server-approved, and thus in DomainTransferRequestFlow we set this field to the
existing computed value serverApproveNewExpirationTime, which is what we use
for setting up the server-approve autorenew billing event and poll message.
In DomainTransferApproveFlow we overwrite this field with the freshly computed
newExpirationTime, whereas in DomainTransferCancel/RejectFlow (and in the
implicit cancel of DomainDeleteFlow during a pending transfer) we null it out.
There are two key benefits to having this field, which are described in more
detail in b/36405140.
1) b/25084229 - it allows storage of a frozen value to back the "exDate" field
of DomainTransferResponse, which we can use to fix various errors with how
exDate display currently works.
2) b/36354434 - it allows DomainResource.cloneProjectedAtTime() to just directly
set the registrationExpirationTime to this value, without computing it de
novo, which reduces duplicated logic and ensures that the new expiration time
matches the autorenew child objects.
This CL only starts writing the field on TransferData as persisted directly on
the DomainResource itself. We'll then want to backfill the field for at
least pending transfers, whether expired or not (so we can do (2) above), but
I think we might as well backfill it for all pending and approved transfers
so that we also fix (1) even for historical transfers. And then we can start
actually reading the field for both purposes. (Note that for (1), this will
only fix synchronous transfer responses served via DomainTransferQueryFlow,
not async transfer responses served via poll messages, since these have already
been persisted with a potentially bad exDate, but I don't think it's worth a
backfill for those).
One last naming note: I chose the verbose transferredRegistrationExpirationTime
rather than the extendedRegistrationExpirationTime of DomainTransferResponse
because (as is the case in autorenew grace, or for a superuser transfer) the
new registration time isn't necessarily extended at all; it may be the same as
the pre-transfer expiration time. Also, including "registration" helps clarify
w.r.t. pendingTransferExpirationTime which refers confusingly to the expiry of
the transfer itself, rather than the domain registration.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171858083
Use a direct reference to the jsch 0.1.53 package instead of going through the
target link, which doesn't exist after MOE transformation. Also change the
target's visibility (since the whitelist referenced by the current rule isn't
preserved by the moe transform).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171554085
TESTED=Verified on Alpha with an updated build using Java 8, Verified on alpha
against the latest java 8 based build.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171297613
Since HistoryEntry always represents a read-only log of mutation to a core resource, that mutation should always happen in a transaction, and the HistoryEntry should be saved in that transaction. As such, it's always more accurate to use ofy().getTransactionTime() for the modificationTime of the HistoryEntry rather than just DateTime.now(UTC).
In addition, having these be the exact same timestamp makes it possible to align HistoryEntries with commit log manifests using modificationTime = transactionTime, which is useful for recovery and analysis purposes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170136957
Mostly based on the original PR, but with some tweaking by nfelt@, in particular to add some support for autorenew grace period subsumption.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169922894
