This allows list_domains to continue working for large TLDs.
TESTED=Deploys to alpha and it works to list the most recently created domains even
on a TLD with a huge number of domains on it (much more than .app has currently).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196717389
This also incorporates general improvements and additions to the existing EPP
lifecycle tests around domain deletion. As a refresher: There is a 5 day
add grace period (AGP) following domain creation. Domains that are deleted
during that period have their create costs (but not EAP costs) refunded. This
deletion takes place immediately. Refunds are implemented by issuing a
Cancellation for the associated OneTime billing event.
Domains that are deleted after AGP ends first go through a 30 day redemption
grace period followed by a 5 day pending deletion period. No create fees are
refunded in this case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179597874
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
I could've sworn we were already doing this, but apparently not? Anyway,
ROID suffixes have a number of requirements on them that weren't being
enforced, so this enforces them. All existing production data is compliant
with these requirements; the only existing bad data we have is in alpha and
sandbox.
ROID suffixes are now required to match the regex ^[A-Z0-9_]{1,8}$
See also https://tools.ietf.org/html/rfc5730
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173400001
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
This CL adds transferredRegistrationExpirationTime as a TransferData field
persisted to Datastore. It's only relevant for domains, and it represents the
registration expiration time resulting from the approval of the most recent
transfer request. For pending transfers, we assume the transfer will be
server-approved, and thus in DomainTransferRequestFlow we set this field to the
existing computed value serverApproveNewExpirationTime, which is what we use
for setting up the server-approve autorenew billing event and poll message.
In DomainTransferApproveFlow we overwrite this field with the freshly computed
newExpirationTime, whereas in DomainTransferCancel/RejectFlow (and in the
implicit cancel of DomainDeleteFlow during a pending transfer) we null it out.
There are two key benefits to having this field, which are described in more
detail in b/36405140.
1) b/25084229 - it allows storage of a frozen value to back the "exDate" field
of DomainTransferResponse, which we can use to fix various errors with how
exDate display currently works.
2) b/36354434 - it allows DomainResource.cloneProjectedAtTime() to just directly
set the registrationExpirationTime to this value, without computing it de
novo, which reduces duplicated logic and ensures that the new expiration time
matches the autorenew child objects.
This CL only starts writing the field on TransferData as persisted directly on
the DomainResource itself. We'll then want to backfill the field for at
least pending transfers, whether expired or not (so we can do (2) above), but
I think we might as well backfill it for all pending and approved transfers
so that we also fix (1) even for historical transfers. And then we can start
actually reading the field for both purposes. (Note that for (1), this will
only fix synchronous transfer responses served via DomainTransferQueryFlow,
not async transfer responses served via poll messages, since these have already
been persisted with a potentially bad exDate, but I don't think it's worth a
backfill for those).
One last naming note: I chose the verbose transferredRegistrationExpirationTime
rather than the extendedRegistrationExpirationTime of DomainTransferResponse
because (as is the case in autorenew grace, or for a superuser transfer) the
new registration time isn't necessarily extended at all; it may be the same as
the pre-transfer expiration time. Also, including "registration" helps clarify
w.r.t. pendingTransferExpirationTime which refers confusingly to the expiry of
the transfer itself, rather than the domain registration.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171858083
Also fixed a bug discovered as a result of the more complete testing!
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171156304
This CL changes the domain and contact transfer flows to check the entire
TransferData on the post-transfer resource, rather than just spot-checking
certain fields. This approach provides much better code coverage - in
particular, it checks that the non-request flows (approve, cancel, reject)
don't modify the fields that they shouldn't be modifying, and that they do
actually clear out the transfer server-approve entities fields written by
the transfer request flow. It's slightly orthogonal, but I also added
testing that the server-approve entities fields are actually set in the
request flows, which was previously untested.
This is pre-work for introducing an exDate-storing field into TransferData,
by making it easier to test everywhere that exDate is set *and* unset only
in the correct places.
As part of this CL, I've introduced a TransferData.copyConstantFieldsToBuilder()
method that is like asBuilder() but instead of copying all the fields to the new
builder, it only copies the logically constant ones: losing/gaining client IDs,
the request time and TRID, and transferPeriod. This is useful both in tests but
is also used in the resolvingPendingTransfer() helper that centralizes the core
transfer resolution logic (as of [] That method has its own tests,
and in the process I removed a bunch of crufty defunct TransferData tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171053454
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).
This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
After this point all data is migrated to use the new canonical
plural version, and subsequent code changes can be made that use
multiple writers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161673486
Memcache is already off but now it's not in the code anymore.
This includes removing domain creation failfast, since that is actually
slower now than just running the flow - all you gain is a non-transactional
read over a transactional read, but the cost is that you always pay that
read, which is going to drive up latency.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158183506
The absence of these fields causes RDE failures, so they are in effect
required on any functioning registry system. We are currently
experiencing problems in sandbox caused by null values on these fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155474895
No tests break because of this, but some tests that were previously
intentionally clearing memcache (for example, the test in
DomainInfoFlow that counts datastore calls) are now doing something
that is not a no-op anymore.
Also fix an incorrect comment about memcache in DatastoreHelper.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154087182
Now that transfers are always restricted to 1 year, it's unnecessary to store
extendedRegistrationYears on TransferData - it will always be equal to 1. This
simplifies logic in a few other places, e.g. RdeDomainImportAction.
I verified in BigQuery that no DomainBases exist with extendedRegistrationYears
values that aren't either null or equal to 1. At some point we should remove
the persisted fields from datastore via e.g. resaving all those domains, but
it's low priority and can wait until we have some more pressing migration.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150373897
EPP host spec section 3.2.5 requires that attempts to update hosts that are linked to different registrars must fail with error code 2305. This is complicated to do, as linked status is eventually consistent, and even more painful when checking links to those of different registrars.
This change forbids external-to-anything renames entirely.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150336754
In fact, completely eviscerate cloneProjectedAtTime (to be removed in
a followup CL) in favor of doing the projection of transfers and the
loading of values from the superordinate domain at call sites. This
is one of the issues that blocked the memcache audit work, since the
load inside of cloneProjectedAtTime could not be controlled by the
caller.
Note: fixed a minor bug where a subordinate host created after its superordinate domain was last transferred should have lastTransferTime==null but was previously reporting the domain's lastTransferTime.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149769125
Principally, this moves a load method into DatastoreHelper that is now
only used by tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148649087
It was kind of messy having all of that logic living alongside the
entities themselves.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148498024
This also cleans up the PremiumList API so that it only has one
method for checking premium prices, which is by TLD, rather than two.
I will be refactoring a lot of the static methods currently residing in
the PremiumList class into a separate utils class, but I don't want to
include too many changes in this one CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148475345
This is the first step in the migration to remove the need to load all of
the premium list entries every time the cache expires (which causes slow-
downs). Once this is deployed, we can re-save all premium lists, creating
the bloom filters, and then the next step will be to read from them to
more efficiently determine if a label might be premium.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=147525017
This is probably best from a code-cleanliness perspective anyways,
but the rationale is that tightly coupling the resources to the
info responses was a straightjacket that required all status
values and fields to be directly available on the resource. With
this change, I already was able to get rid of the preMarshal()
hackery, and I will be able to get rid of cloneWithLinkedStatus()
and most of the contents of cloneProjectedAtTime() for non-domains.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=144252924
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
The next step will be to get rid of RegistryConfig descendants and RegistryConfigLoader entirely.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143812815
This is the first in a decently long series of commits to delete RegistryConfig
entirely and centralize all configuration in ConfigModule using Dagger. Once
this is done, then the text-based YAML configuration work can begin in earnest.
Note that the configuration settings from TestRegistryConfig will be moving
into ConfigModule.LocalTestConfig. This way they can be referred to in a static
context from test and test utility helpers, rather than having to be injected
everywhere, which we don't typically bother with for tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143473089
Also adds some clarity on when to use the non-incrementing version.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143037508
It was already doing this for domains and hosts, and that it wasn't doing it for
contacts was confusing and caused me to experience a difficult-to-track-down
error in writing new code.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134827468
They were taking a DateTime "now", which would seem like it would be the time of
when the resource was deleted, but it was actually the time by which the
resource was deleted, with the actual deletion time being hardcoded to a day
prior. The confusion was evident because a fair number of tests were passing
the wrong thing. I renamed the parameter "deletionTime" to make it exactly
clear what it's doing and fixed up some callsites where necessary.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134818032
This will replace the existing DnsRefreshForHostRenameAction.
This is stage one of a three stage migration process. It adds the new queue and
[] but doesn't call them yet. Stage two will cut over to using the new
functionality, and stage three will remove the old functionality.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134793963
It appears to be standard RDAP practice when returning result sets for domain, nameserver and entity searches to give only summary data for each result item. Any information that can be gleaned from the object itself is included, but related resources are not included. For a domain, for instance, the domain information is included, but nameservers, entities and events (which come from history entries) are suppressed. In their place, there is a standard boilerplate remark in the object indicating that only summary data is included, and that the user should query the item directly to get the full data. Note that summary data is used only for searches; direct queries for an item will still return full data.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133973835
It's best to be consistent and use the same thing everywhere. "clientId" was
already used in more places and is shorter and no more ambiguous, so it's the
logical one to win out.
Note that this CL is almost solely a big Eclipse-assisted refactoring. There are
two places that I did not change clientIdentifier -- the actual entity field on
Registrar (though I did change all getters and setters), and the name of a
column on the exported registrar spreadsheet. Both would require data
migrations.
Also fixes a few minor nits discovered in touched files, including an incorrect
test in OfyFilterTest.java and some superfluous uses of String.format() when
calling checkArgument().
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133956465
When EAP is involed we current have one billing event for domain create that
has the create fee and EAP fee lumped together. Change it to record two
separate billing events for each.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132335349
This change replaces all Ref objects in the code with Key objects. These are
stored in datastore as the same object (raw datastore keys), so this is not
a model change.
Our best practices doc says to use Keys not Refs because:
* The .get() method obscures what's actually going on
- Much harder to visually audit the code for datastore loads
- Hard to distinguish Ref<T> get()'s from Optional get()'s and Supplier get()'s
* Implicit ofy().load() offers much less control
- Antipattern for ultimate goal of making Ofy injectable
- Can't control cache use or batch loading without making ofy() explicit anyway
* Serialization behavior is surprising and could be quite dangerous/incorrect
- Can lead to serialization errors. If it actually worked "as intended",
it would lead to a Ref<> on a serialized object being replaced upon
deserialization with a stale copy of the old value, which could potentially
break all kinds of transactional expectations
* Having both Ref<T> and Key<T> introduces extra boilerplate everywhere
- E.g. helper methods all need to have Ref and Key overloads, or you need to
call .key() to get the Key<T> for every Ref<T> you want to pass in
- Creating a Ref<T> is more cumbersome, since it doesn't have all the create()
overloads that Key<T> has, only create(Key<T>) and create(Entity) - no way to
create directly from kind+ID/name, raw Key, websafe key string, etc.
(Note that Refs are treated specially by Objectify's @Load method and Keys are not;
we don't use that feature, but it is the one advantage Refs have over Keys.)
The direct impetus for this change is that I am trying to audit our use of memcache,
and the implicit .get() calls to datastore were making that very hard.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131965491
This is one of several CLs in a sequence for allowing per-TLD DNS
implementations.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129445641
This is better than the previous way of using the canonical name of the class,
because the previous way did not allow for refactoring, and also required the
PremiumPricingEngine to live in the model package lest there be circular
dependencies, which does not seem ideal.
Note that, for reasons of backwards compatibility with existing persisted data,
the name of the static premium pricing engine has been set to its canonical
class name, but the class can now be refactored going forward so long as this
string remains unchanged, and any new pricing engine implementations can use
whatever string key they want (it doesn't have to be a canonical class name).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129215185
The presubmits are warning that toUpperCase() and toLowerCase() are locale-specific, and advise using Ascii.toUpperCase() and Ascii.toLowerCase() as a local-invariant alternative.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=127583677
