Also added a couple of more tests to make sure that we cover all edge cases.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195872013
Currently we determine fee type from the fee extension description by checking if the format string of the FeeType contains the description we received. The formatting string for EAP is "Early Access Period, fee expires: %s", so the fee description generated by a domain check command, like "Early Access Period, fee expires: 2022-03-01T00:00:00.000Z", is not recognized as EAP.
This CL adds the ability to add arbitrary extra description strings to the FeeType for a description to match against. It also changes the match to "the given description contains any of the strings from the list of format string plus extra description strings".
For EAP, we added an extra description string "Early Access Period", so any fee description that contains "Early Access Period" will be matched to EAP FeeType, including the specific description (that contains the expiry time) that we send in a domain check.
Also improved error message on multiple fee type matching.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194149162
This verifies anchor tenant creates by registrars are for exactly 2 years, to simplify billing down the line (anchor tenants get 2 years of free domain creates).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193931858
This allows a registrar to check domain availability even if the fee extension is not set. To obtain the premium price or create the domain, the fee extension is still needed.
PS: I believe the previous tests are erroneous anyway. It is the presence of fee extensions in the check request that we were looking for. Checking if the extension is declared during login has no bearing on the results.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192778137
A "mark" tells us that the holder owns the trademark for a given domain name. It is signed for authentication.
If the signature's certificate is either "not yet valid" or "expired", we return explicit errors to that effect.
But in addition to the signature's certificate, the mark itself might not be valid yet or already expired. Right now if that happens - we return an error saying "the mark doesn't match the domain name".
That is wrong - as the mark can match the domain name, just be expired. Returning "the mark doesn't match the domain name" in that case is misleading.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190069976
This simplifies calculating the overall invoice by giving RESTORE fees a
period equal to the period of the associated RENEW (1 year). Older
BillingEvents will not be backfilled, and will have periodYears = null.
Invoicing and business both agree this is a valid representation, since RESTORE fees are intrinsically tied to the 1-year RENEW it's associated with.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=188041777
It was nullable all along, but wasn't tagged as such, and thus it was
possible to misuse the method from its call sites.
Also adds an assertion about no NORDN tasks being enqueued in a failing
domain create test for a required signed mark.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187649865
Also changed the name of "verifyRegistryStateAllowsLaunchFlows" to "verifyRegistryStateAllowsApplicationFlows", because there are now launch flows that don't use applications (start-date sunrise).
Finally, added a test to showcase the "super-user" power that EPPs with Anchor Tenants have. There's no change in behavior in that regard in this CL - we just add a test to make it explicit.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187517199
After investigating common domain create/update command usage
patterns by registrars, we noticed that it is frequent for a
given registrar to reuse both hosts (using a standardized set of
nameservers) as well as contacts (e.g. for privacy/proxy
services). With these usage patterns, potential per-registrar
throughput during high volume scenarios (i.e. first moments of
General Availability) suffers from hitting hot keys in Datastore.
The solution, implemented in this CL, is to add short-term
in-memory caching for contacts and hosts, analogous to how we are
already caching Registry and Registrar entities. These new
cached paths are only used inside domain flows to determine
existence and deleted/pending delete status of contacts and
hosts. This is a potential loss of transactional consistency, but
in practice it's hard to imagine this having negative effects, as
contacts or hosts that are in use cannot be deleted, and caching
would primarily affect widely used contacts and hosts.
Note that this caching can be turned on or off through a
configuration option, and by default would be off. We'd only want
it on when we really needed it, i.e. during a big launch.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187093378
I'm actually surprised that we had this in our code, as it seems like a huge
oversight, but we were individually loading each and every referenced contact
and host during domain/application create/update/allocate flows. Loading them
all as a single batch should reduce round trips to Datastore by a good deal,
thus improving performance.
We aren't giving up any transactional consistency in doing so and the only
potential downside I can think of is that we're always loading all contacts/
hosts instead of only some of them, in the rare case that one of the earlier
contacts/hosts is actually in pending delete (which allowed short-circuiting).
However, the gains from only making one round-trip should swamp the potential
losses in occasionally loading more data than is strictly necessary.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=186656118
Currently we validate the fee extension by summing up all fees present in the extension and comparing it against the total fee to be charged. While this works in most cases, we'd like the ability to individually validate each fee. This is especially useful during EAP when two fees are charged, a regular "create" fee that would also be amount we charge during renewal, and a one time "EAP" fee.
Because we can only distinguish fees by their descriptions, we try to match the description to the format string of the fee type enums. We also only require individual fee matches when we are charging more than one type of fees, which makes the change compatible with most existing use cases where only one fees is charged and the description field is ignored in the extension.
We expect the workflow to be that a registrar sends a domain check, and we reply with exactly what fees we are expecting, and then it will use the descriptions in the response to send us a domain create with the correct fees.
Note that we aggregate fees within the same FeeType together. Normally there will only be one fee per type, but in case of custom logic there could be more than one fee for the same type. There is no way to distinguish them as they both use the same description. So it is simpler to just aggregate them.
This CL also includes some reformatting that conforms to google-java-format output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=186530316
The START_DATE_SUNRISE phase allows registration of domains only with a signed mark. In all other respects - it is identical to the GENERAL_AVAILABILITY phase.
Note that Anchor Tenants bypass all checks, and are hence able to register domains without a signed mark.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=185534793
When enabled for a registrar, all EPP operations on premium domains that have
costs (e.g. creates, renews, transfers) will fail unless the EPP fee extension
is used to explicitly ack the amount of fee as part of the EPP transaction.
This ack is required regardless of whether premium fee acking is required at
the registry level. No data migration is necessary since false is the desired
default for this new attribute.
This CL also contains some slight refactoring of static utility methods used to
perform fee verification; there was short-circuiting at call-sites in two
places when what was really needed was two methods, one implementing additional
functionality on top of the other, and calling the inner method in the places
where short-circuiting had previously been necessary.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=184229363
This uses an extensibility mechanism similar to that of WhoisCommandFactory
and CustomLogicFactory, namely, that a fully qualified Java class is
specified in the YAML file for each environment with the allocation token
custom logic to be used. By default, this points to a no-op base class
that does nothing. Users that wish to add their own allocation token
custom logic can simply create a new class that extends
AllocationTokenCustomLogic and then configure it in their .yaml config
files.
This also renames the existing *FlowCustomLogic *Flow instance variables
from customLogic to flowCustomLogic, to avoid the potential confusion with
the new AllocationTokenCustomLogic class that also now exists.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=183003112
Next up is adding custom logic so that the results of these checks can be
more meaningful.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181660956
The next step is to add them for domain checks as well (which is simpler
because it doesn't involve validation).
This requires the addition of a TrimWhitespaceAdapter for XML JAXB objects,
which will prove useful for other @XmlValue attributes in the future.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181526726
The extension itself isn't used for anything yet; allocation tokens can be
passed for domain creates and checks but are ignored if present. This will
be changed in a subsequent CL that adds AllocationToken entities and related
logic. Usage of this extension in any other EPP flow will throw an
UnsupportedExtensionException.
The relevant spec is https://tools.ietf.org/html/draft-gould-allocation-token-04
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181343188
Stream.concat only accepts 2 parameters. Streams.concat on the other hand
accepts any number of parameters.
Moving to Streams.concat for all uses (2 or more) makes sense for uniformity
and convenience reasons.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179716648
Failing to use the fee extension during EAP can result in charges to registrars
that are radically different than what they may have been expecting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177597883
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
This command is used by registry operators to apply registry locks to
domain names.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176549874
This way it is consistent with the rest of our domain-related flows, which
consistently use the Domain* prefix. Note that claims checks are just a
special case of domain checks anyway, which run under DomainCheckFlow. This
will make dashboards looking at domain commands "just work" with a regexp of
Domain.*, without having to special-case in ClaimsCheck.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172608964
The nameserver may be external, in which case its TLD will not appear in our
list of valid TLDs, and the search will be rejected erroneously.
Tests for letter case canonicalizations also added at reviewer's suggestion.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171985702
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
This CL adds transferredRegistrationExpirationTime as a TransferData field
persisted to Datastore. It's only relevant for domains, and it represents the
registration expiration time resulting from the approval of the most recent
transfer request. For pending transfers, we assume the transfer will be
server-approved, and thus in DomainTransferRequestFlow we set this field to the
existing computed value serverApproveNewExpirationTime, which is what we use
for setting up the server-approve autorenew billing event and poll message.
In DomainTransferApproveFlow we overwrite this field with the freshly computed
newExpirationTime, whereas in DomainTransferCancel/RejectFlow (and in the
implicit cancel of DomainDeleteFlow during a pending transfer) we null it out.
There are two key benefits to having this field, which are described in more
detail in b/36405140.
1) b/25084229 - it allows storage of a frozen value to back the "exDate" field
of DomainTransferResponse, which we can use to fix various errors with how
exDate display currently works.
2) b/36354434 - it allows DomainResource.cloneProjectedAtTime() to just directly
set the registrationExpirationTime to this value, without computing it de
novo, which reduces duplicated logic and ensures that the new expiration time
matches the autorenew child objects.
This CL only starts writing the field on TransferData as persisted directly on
the DomainResource itself. We'll then want to backfill the field for at
least pending transfers, whether expired or not (so we can do (2) above), but
I think we might as well backfill it for all pending and approved transfers
so that we also fix (1) even for historical transfers. And then we can start
actually reading the field for both purposes. (Note that for (1), this will
only fix synchronous transfer responses served via DomainTransferQueryFlow,
not async transfer responses served via poll messages, since these have already
been persisted with a potentially bad exDate, but I don't think it's worth a
backfill for those).
One last naming note: I chose the verbose transferredRegistrationExpirationTime
rather than the extendedRegistrationExpirationTime of DomainTransferResponse
because (as is the case in autorenew grace, or for a superuser transfer) the
new registration time isn't necessarily extended at all; it may be the same as
the pre-transfer expiration time. Also, including "registration" helps clarify
w.r.t. pendingTransferExpirationTime which refers confusingly to the expiry of
the transfer itself, rather than the domain registration.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171858083
In the great flow flattening, ResourceFlowUtils grew a couple nice helpers
for rebuilding transferrable resources (Domains and Contacts) upon the
resolution of a transfer - approvePendingTransfer() and denyPendingTransfer().
Most transfer-resolving callsites use one of these two helpers, but for legacy
reasons the deletion flows (DomainDeleteFlow and DeleteContactsAndHostsAction)
were instead using the "manual" resolvePendingTransfer() method or its even more
low-level createResolvedTransferData() helper instead of denyPendingTransfer().
It's simpler to just have two options - approve and deny - so this CL inlines
createResolvedTransferData() into resolvePendingTransfer() and makes the latter
a private helper for the approve/denyPendingTransfer() public helpers.
This CL also adds sanity checks that approve/denyPendingTransfer() are called
only with the logically appropriate values of TransferStatus.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170819358
Normally, if a domain is in the auto-renew grace period, a transfer will cancel the auto-renew billing event. In the event of a transfer with no change to registration end date, the auto-renew billing event should not be cancelled and the gaining registrar should not be charged for the transfer.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170576726
Specifically, this prevents suspended registrars from creating domains or applications. Pending registrars already can't perform these actions because they get an error message when attempting to log in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170481338
The bad prober domain data has since been deleted, so we no longer need
to handle the case where these Keys point to entities that don't exist.
This mostly reverses []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168687701
Allow superusers to change the grace period and allow
superusers to change the pending delete length.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168028545
Allow superusers to change the transfer period to zero years and allow
superusers to change the automatic transfer length.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598314
Previously, I would cancel all the records associated with HistoryEntry that's
available for cancellation. This could cause unexpected behavior if we
cancelled a historyEntry which itself had cancelled records (in effect we would
negate the negation unintentionally). This is easily remedied by only
cancelling records which want to be cancelled.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167204383
The probers make a constant stream of create and delete calls, which we don't
want to account for when constructing transaction reports. This change will
cause only real TLDs to log create and delete transaction records.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166737801
This is the last of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
The transfer process is as follows:
GAINING sends a TransferRequest to LOSING
LOSING either acks (TransferApprove), nacks (TransferReject) or does nothing
(auto approve). For acks and autoapproves, we produce a +1 counter for GAINING
and LOSING for domain-gaining/losing-successful for each registrar, to be
reported on the approve date + the transfer grace period. For nacks, we produce
a +1 counter for domain-gaining/losing-nacked for each registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166535579
This is the third of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
We add a +1 counter for either grace or nograce deletes, based on the grace period status of the domain. We then search back in time for DOMAIN_CREATE, DOMAIN_RENEW and DOMAIN_AUTORENEW HistoryEntries off the same resource that happened in their corresponding grace periods (5, 5 and 45 days respectively). All transaction records for these events are then given -1 counters to properly account for cancellations in the NET_CREATE and NET_RENEW fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166506010
After working further with domain deletes, I realized we'll need to record multiple reportingTimes under a single historyEntry when issuing a -1 counter to cancel grace-period adds. Since the TLD would be the only shared component within a record, we'll just duplicate it across all records to save an unnecessary layer of hierarchy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166261413
This is the second of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
Adds and renews each result in a +1 counter for the NET_ADDS/RENEWS_#_YR field,
which I've added simple (# of years, add or renew) -> Enum functions to get.
Allocates are just a special case of adds, and are counted in a similar manner.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165963249
This is the first of many cls adding explicit logging in all our domain mutation flows to facilitate transaction reporting.
Restores are relatively simple- it happens immediately, so the reporting time is just the time of the HistoryEntry, and we add a single "RESTORED_DOMAINS" count of 1.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165639084
Added validation on domain creation, preventing a domain from being created if
it equals an existing TLD. Added domain create tests for domains using
multi-part TLDs that shared suffixes and prefixes. Added host create tests for
hosts using multi-part TLDs that shared suffixes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164297749
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).
This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
