Commit graph

135 commits

Author SHA1 Message Date
mcilwain
2aa897e698 Remove unnecessary generic type arguments
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175155365
2017-11-21 18:17:31 -05:00
mcilwain
eed2e0c45f Remove unnecessary explicit generic type declarations
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
2017-11-07 17:11:29 -05:00
mcilwain
1790914058 Add Runnable overrides to ease use of Java 8 language features
Runnable and Callable are both @FunctionalInterfaces. The difference is
that Callable requires a return value whereas Runnable does not, so in
situations where we don't care about a return value, rather than having to
add an unnecessary 'return null;' at the end of the lambda, we can simply
use a non-returning Runnable instead.

Unfortunately, owing to legacy reasons, Runnable is not declared to throw
checked exceptions whereas Callable is, so in situations where checked
exceptions are thrown we still need to have a 'return null;' call at the
end.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172935400
2017-10-24 16:53:47 -04:00
guyben
06f0ec4f2f Remove reference to TLD in Locks' ResourceName
The TLD is added separately to the lock (the lock is unique per
TLD-ResourceName pair), so there's no need to add it to the resourceName.

The current status is that the TLD was included twice in the lockId. After the fix - it'll only be included once.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172659986
2017-10-24 16:53:47 -04:00
mcilwain
05f6ad80ab Format some imports using fix_imports.py
Unfortunately this tool isn't smart enough to deal with the assertThat
situation (which has two static imports of a function with the same name).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172000753
2017-10-24 16:53:47 -04:00
mcilwain
c0f8da0c6e Switch from Guava Optionals to Java 8 Optionals
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
2017-10-24 16:53:47 -04:00
nickfelt
184b2b56ac Persist transferredRegistrationExpirationTime (exDate) on TransferData
This CL adds transferredRegistrationExpirationTime as a TransferData field
persisted to Datastore.  It's only relevant for domains, and it represents the
registration expiration time resulting from the approval of the most recent
transfer request.  For pending transfers, we assume the transfer will be
server-approved, and thus in DomainTransferRequestFlow we set this field to the
existing computed value serverApproveNewExpirationTime, which is what we use
for setting up the server-approve autorenew billing event and poll message.
In DomainTransferApproveFlow we overwrite this field with the freshly computed
newExpirationTime, whereas in DomainTransferCancel/RejectFlow (and in the
implicit cancel of DomainDeleteFlow during a pending transfer) we null it out.

There are two key benefits to having this field, which are described in more
detail in b/36405140.

1) b/25084229 - it allows storage of a frozen value to back the "exDate" field
   of DomainTransferResponse, which we can use to fix various errors with how
   exDate display currently works.

2) b/36354434 - it allows DomainResource.cloneProjectedAtTime() to just directly
   set the registrationExpirationTime to this value, without computing it de
   novo, which reduces duplicated logic and ensures that the new expiration time
   matches the autorenew child objects.

This CL only starts writing the field on TransferData as persisted directly on
the DomainResource itself.  We'll then want to backfill the field for at
least pending transfers, whether expired or not (so we can do (2) above), but
I think we might as well backfill it for all pending and approved transfers
so that we also fix (1) even for historical transfers.  And then we can start
actually reading the field for both purposes.  (Note that for (1), this will
only fix synchronous transfer responses served via DomainTransferQueryFlow,
not async transfer responses served via poll messages, since these have already
been persisted with a potentially bad exDate, but I don't think it's worth a
backfill for those).

One last naming note: I chose the verbose transferredRegistrationExpirationTime
rather than the extendedRegistrationExpirationTime of DomainTransferResponse
because (as is the case in autorenew grace, or for a superuser transfer) the
new registration time isn't necessarily extended at all; it may be the same as
the pre-transfer expiration time.  Also, including "registration" helps clarify
w.r.t. pendingTransferExpirationTime which refers confusingly to the expiry of
the transfer itself, rather than the domain registration.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171858083
2017-10-24 16:53:47 -04:00
mmuller
69d33e928a Use a direct reference to jsch 0.1.53
Use a direct reference to the jsch 0.1.53 package instead of going through the
target link, which doesn't exist after MOE transformation.  Also change the
target's visibility (since the whitelist referenced by the current rule isn't
preserved by the moe transform).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171554085
2017-10-10 12:09:42 -04:00
mmuller
a790ec27db Upgrade domain-registry to jsch 0.1.53
TESTED=Verified on Alpha with an updated build using Java 8, Verified on alpha
against the latest java 8 based build.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171297613
2017-10-10 12:09:42 -04:00
mcilwain
5edb7935ed Run automatic Java 8 conversion over codebase
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171174380
2017-10-10 12:09:41 -04:00
nickfelt
2702c3a7f3 Fix code to always set HistoryEntry modification time using txn time
Since HistoryEntry always represents a read-only log of mutation to a core resource, that mutation should always happen in a transaction, and the HistoryEntry should be saved in that transaction.  As such, it's always more accurate to use ofy().getTransactionTime() for the modificationTime of the HistoryEntry rather than just DateTime.now(UTC).

In addition, having these be the exact same timestamp makes it possible to align HistoryEntries with commit log manifests using modificationTime = transactionTime, which is useful for recovery and analysis purposes.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170136957
2017-10-04 16:16:45 -04:00
nickfelt
3ad21e3834 Fix RDE import pending transfer handling
Mostly based on the original PR, but with some tweaking by nfelt@, in particular to add some support for autorenew grace period subsumption.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169922894
2017-10-04 16:16:45 -04:00
mcilwain
8bd9b35dfd Link domain to subordinate hosts on RDE import
Based on the original pull request below with some modifications for code drift
over time (including adding handling/testing for the case where superordinate
domains are in pending delete, and thus can't have hosts linked to them).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169698157
2017-10-04 16:16:45 -04:00
guyben
3cc44259da Use different locks for thin and full rde reports
The lock is used to prevent dual writing of the same output file, and to
prevent us from generating the same data twice (if RdeStaging was run twice).

However, at times when we're creating both RDE and BRDA reports - we use the
same lock for both even though they write to different files and create
different reports.

This causes one of the two to almost always fail (when both are created, which
is once a week I think) delaying that report by several hours.

Note that the "prefix" part of the filenames has the mode in it, so different
modes will not have the same files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168882007
2017-09-20 10:27:17 -04:00
guyben
c3861f6e95 Swap all uses of Lock to LockHandler
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167661348
2017-09-12 15:51:50 -04:00
bbilbo
2e4b63bb79 Add support for a domain transfer request superuser EPP extension
Allow superusers to change the transfer period to zero years and allow
superusers to change the automatic transfer length.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598314
2017-09-12 15:51:50 -04:00
mcilwain
103b3d7608 Correct usages of DateTime.now() everywhere to DateTime.now(UTC)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166393128
2017-08-29 17:09:53 -04:00
guyben
aee4f7acc2 Remove queueing from Lock
It was buggy (didn't work) and was never actually used.

Why never actually used: for it to be used executeWithLock has to be called
with different requesters on the same lockId. That never happend in the code.

How it was buggy: Logically, the queue is deleted on release of the lock (meaning it was
meaningless the only time it mattered - when the lock isn't taken). In
addition, a different bug meant that having items in the queue prevented the
lock from being released forcing all other tasks to have to wait for lock
timeout even if the task that acquired the lock is long done.

Alternative: fix the queue. This would mean we don't want to delete the lock on release (since we want to keep the queue). Instead, we resave the same lock with expiration date being START_OF_TIME. In addition - we need to fix the .equals used to determine if the lock the same as the acquired lock - instead use some isSame function that ignores the queue.

Note: the queue is dangerous! An item (calling class / action) in the first place of a queue means no other calling class can get that lock. Everything is waiting for the first calling class to be re-run - but that might take a long time (depending on that action's rerun policy) and even might never happen (if for some reason that action decided it was no longer needed without acquiring the lock) - causing all other actions to stall forever!

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163705463
2017-08-01 17:06:20 -04:00
mcilwain
d536cef20f Make Registrar load methods return Optionals instead of Nullables
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).

This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
2017-08-01 16:58:59 -04:00
guyben
e224a67eda Change @Auth to an AutoValue, and created a set of predefined Auths
We want to be safer and more explicit about the authentication needed by the many actions that exist.

As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
2017-08-01 16:33:10 -04:00
mountford
009522d078 Change auth settings on RdeStagingAction
[] opened up the settings on RdeStaging, in order to make it usable by the nomulus tool. But in retrospect, we think that all we needed to do was support the POST method, not loosen the auth settings, since nomulus invokes RdeStaging via a task queue. Removing the looser auth settings will bring this action into line with other backend actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157262595
2017-06-05 18:17:09 -04:00
mountford
7d5a67f5a2 Localize a variable
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156187775
2017-05-17 12:27:43 -04:00
cgoldfeder
c9d7e75946 Cache Registrars in memory
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.

While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
2017-05-17 12:26:11 -04:00
mountford
74a0defef3 Add the ability to generate RDE deposits in lenient mode
We will not want to run this under normal circumstances, but in cases (such as PDT testing in sandbox) where it's desirable to generate an escrow deposit even if it isn't technically valid XML, this gives us that option. Manual-mode RDE generation is also changed so that, if no watermark date is specified, it defaults to the previous midnight, to better support running of RDE in sandbox to catch data problems.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156108295
2017-05-17 12:22:49 -04:00
guyben
93c2a1e4f0 Make JSch injection lazy
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154958750
2017-05-17 11:22:53 -04:00
nickfelt
7f2821cd7e Open up RdeStagingAction @Action settings to support tool access
This change means that RdeStagingAction now accepts POST requests and supports the API auth method, which is needed for the new GenerateEscrowDepositCommand (as of [] to be able to invoke the action in manual mode.  Note that this shouldn't affect the cron setting that invokes it currently, which are GET requests with INTERNAL auth.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154764870
2017-05-03 11:09:29 -04:00
mountford
11e7374c0f Fix bug which allowed creation of hosts with superordinate domains in pending delete state.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154716883
2017-05-03 11:03:23 -04:00
mountford
dea386d08a Change GenerateEscrowDepositCommand to trigger back end deposit generation
Previously, GenerateEscrowDepositCommand generated the deposit itself. Channeling it through the existing deposit generation code make things more maintainable.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152847950
2017-04-13 10:57:24 -04:00
mcilwain
8653d2f204 Refactor out creation of server TRIDs so they can be tested
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152837185
2017-04-13 10:55:47 -04:00
mountford
4f94464eaf Allow RdeStagingAction to be invoked manually
RdeStagingAction always processed all RDE and BRDA deposits currently outstanding, updating the cursors appropriately and kicking off the upload job. Sometimes we don't want all that. We just want to create a specific deposit by hand, without modifying the cursors or uploading. This CL adds parameters to support that.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152415959
2017-04-10 13:47:51 -04:00
nickfelt
09f619cce2 Remove obsolete TransferData.extendedRegistrationYears
Now that transfers are always restricted to 1 year, it's unnecessary to store
extendedRegistrationYears on TransferData - it will always be equal to 1.  This
simplifies logic in a few other places, e.g. RdeDomainImportAction.

I verified in BigQuery that no DomainBases exist with extendedRegistrationYears
values that aren't either null or equal to 1.  At some point we should remove
the persisted fields from datastore via e.g. resaving all those domains, but
it's low priority and can wait until we have some more pressing migration.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150373897
2017-03-21 15:35:00 -04:00
cgoldfeder
9174855a47 Remove the ofy().load() inside of HostResource.cloneProjectedAtTime
In fact, completely eviscerate cloneProjectedAtTime (to be removed in
a followup CL) in favor of doing the projection of transfers and the
loading of values from the superordinate domain at call sites. This
is one of the issues that blocked the memcache audit work, since the
load inside of cloneProjectedAtTime could not be controlled by the
caller.

Note: fixed a minor bug where a subordinate host created after its superordinate domain was last transferred should have lastTransferTime==null but was previously reporting the domain's lastTransferTime.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149769125
2017-03-13 11:22:55 -04:00
Ben McIlwain
2bba5660c2 Use Immutable types for static final fields
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149172966
2017-03-07 13:50:53 -05:00
mcilwain
64bfbea324 Remove duplicate version of lookupSuperordinateDomain() function
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148657943
2017-03-07 13:12:09 -05:00
Wolfgang Meyers
9f90597691 Handle pending transfers on domain imports from RDE deposits
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148496597
2017-02-27 11:30:40 -05:00
mcilwain
cdadb54acd Refer to Datastore everywhere correctly by its capitalized form
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=147479683
2017-02-17 12:12:12 -05:00
Wolfgang Meyers
da1f83a7b4 Add host linker mapreduce for RDE imports
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=147025088
2017-02-09 14:24:27 -05:00
Wolfgang Meyers
6cdb3d81d3 Generate authInfo for imported domains
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146956307
2017-02-08 17:42:57 -05:00
Wolfgang Meyers
60a3d2a53b Synchronize domains with DNS on import
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146813124
2017-02-08 17:42:43 -05:00
Wolfgang Meyers
a904f2c6ee Add mapper to import domains from RDE deposits
With some additional cleanup by Ben McIlwain.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146695665
2017-02-06 16:46:46 -05:00
cgoldfeder
1577ab2c26 Remove an unused exception
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146258125
2017-02-02 17:04:06 -05:00
mmuller
b70f57b7c7 Update copyright year on all license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146111211
2017-02-02 16:27:22 -05:00
cgoldfeder
0b1781b110 Make LINKED into a virtual status value
* Remove LINKED when loading an EppResource
* Enforce that you can't add it to a resource
* Ignore LINKED on xjc import of contacts and hosts

After running ResaveAllEppResourcesAction we will no
longer have persisted LINKED statuses in datastore.

In the process of writing this I discovered that RDAP
treats LINKED like any other status value and returns
the persisted value rather than the derived one. Since
this is an existing bug and is orthogonal to the changes
in this CL, I am addressing it in a separate CL.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145585227
2017-01-30 15:03:53 -05:00
mcilwain
eaec03e670 Move ConfigModule and LocalTestConfig into RegistryConfig
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
2017-01-09 12:01:09 -05:00
mcilwain
c5c74961bb Make all but one field on RegistryConfig static
The next step will be to get rid of RegistryConfig descendants and RegistryConfigLoader entirely.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143812815
2017-01-09 12:01:09 -05:00
jart
734130aa73 Restructure Maven dependencies in build
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072

We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018

A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55

The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.

Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:

- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
2017-01-09 11:59:04 -05:00
mcilwain
28f6c770c8 Add MOE equivalence for sync on 2016-12-19
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142449539
2017-01-09 11:59:04 -05:00
mcilwain
a56a0677ba Refactor RDE import stuff into its own rde.imports package
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141566236
2016-12-13 15:57:46 -05:00
mountford
50aad72383 Add retry logic to RDE upload
We experience intermittent exceptions when connecting to Iron Mountain. Just retrying is probably the easiest solution to the problem.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141221179
2016-12-07 15:19:43 -05:00
Wolfgang Meyers
f2faf49d58 Add XjcToDomainResourceConverter for RDE import
With some code cleanup/refactoring/formatting by Ben McIlwain.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141199855
2016-12-07 15:19:29 -05:00