This will allow us to check in actual SUNRISE billing policies per launch (15% discount), instead of relying on ad-hoc timestamps.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200077926
Also prevents signed marks from being used in non-sunrise TldStates.
Currently, we send out a Lordn update only when there's a ClaimNotice, or if
we're in end-date sunrise.
But EPPs can contain a SignedMark instead of a ClaimsNotice for trademarked
domains - in which case we aren't sending out Lordn update. This also applies
to start-date sunrises.
We also change the SignedMark behavior for superusers. Currently, if a
mismatched signed mark is given as superuser, we accept it. That causes
problems when we want to send the Lordn update.
Instead - we no longer allow superusers to give a mismatched SignedMark (just
as we don't allow users to give a bad ClaimNotice). A super user can still
create a domain WITHOUT a signed mark - but if one is provided, it must match.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199783411
This is consistent with how we treat RESTORE billing events as well- in
general, fees are considered to be amortized over the course of a year (by the
invoicing team).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199684843
Also added a couple of more tests to make sure that we cover all edge cases.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195872013
Currently we determine fee type from the fee extension description by checking if the format string of the FeeType contains the description we received. The formatting string for EAP is "Early Access Period, fee expires: %s", so the fee description generated by a domain check command, like "Early Access Period, fee expires: 2022-03-01T00:00:00.000Z", is not recognized as EAP.
This CL adds the ability to add arbitrary extra description strings to the FeeType for a description to match against. It also changes the match to "the given description contains any of the strings from the list of format string plus extra description strings".
For EAP, we added an extra description string "Early Access Period", so any fee description that contains "Early Access Period" will be matched to EAP FeeType, including the specific description (that contains the expiry time) that we send in a domain check.
Also improved error message on multiple fee type matching.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194149162
This verifies anchor tenant creates by registrars are for exactly 2 years, to simplify billing down the line (anchor tenants get 2 years of free domain creates).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193931858
A "mark" tells us that the holder owns the trademark for a given domain name. It is signed for authentication.
If the signature's certificate is either "not yet valid" or "expired", we return explicit errors to that effect.
But in addition to the signature's certificate, the mark itself might not be valid yet or already expired. Right now if that happens - we return an error saying "the mark doesn't match the domain name".
That is wrong - as the mark can match the domain name, just be expired. Returning "the mark doesn't match the domain name" in that case is misleading.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190069976
<launch:create> has an optional type argument, that can take either "application" or "registration":
https://tools.ietf.org/html/rfc8334#section-3.3.1
We get that type via createExtension.get().getCreateType(), where if the type= argument isn't given, the function returns null.
In that case, we need to decide based on the TLD - application for end-date sunrise, and registration for start-date sunrise.
For now we can't do that, because FlowPicker doesn't have access to the TLD information. Until that is fixed we decide as follows:
- landrush and sunrush phases will default to APPLICATION, because there's no possible
registration for it.
- sunrise defaults to REGISTRATION because we're currenly launching start-date sunrise that uses registration.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189942568
It was nullable all along, but wasn't tagged as such, and thus it was
possible to misuse the method from its call sites.
Also adds an assertion about no NORDN tasks being enqueued in a failing
domain create test for a required signed mark.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187649865
Also changed the name of "verifyRegistryStateAllowsLaunchFlows" to "verifyRegistryStateAllowsApplicationFlows", because there are now launch flows that don't use applications (start-date sunrise).
Finally, added a test to showcase the "super-user" power that EPPs with Anchor Tenants have. There's no change in behavior in that regard in this CL - we just add a test to make it explicit.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187517199
After investigating common domain create/update command usage
patterns by registrars, we noticed that it is frequent for a
given registrar to reuse both hosts (using a standardized set of
nameservers) as well as contacts (e.g. for privacy/proxy
services). With these usage patterns, potential per-registrar
throughput during high volume scenarios (i.e. first moments of
General Availability) suffers from hitting hot keys in Datastore.
The solution, implemented in this CL, is to add short-term
in-memory caching for contacts and hosts, analogous to how we are
already caching Registry and Registrar entities. These new
cached paths are only used inside domain flows to determine
existence and deleted/pending delete status of contacts and
hosts. This is a potential loss of transactional consistency, but
in practice it's hard to imagine this having negative effects, as
contacts or hosts that are in use cannot be deleted, and caching
would primarily affect widely used contacts and hosts.
Note that this caching can be turned on or off through a
configuration option, and by default would be off. We'd only want
it on when we really needed it, i.e. during a big launch.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187093378
Currently we validate the fee extension by summing up all fees present in the extension and comparing it against the total fee to be charged. While this works in most cases, we'd like the ability to individually validate each fee. This is especially useful during EAP when two fees are charged, a regular "create" fee that would also be amount we charge during renewal, and a one time "EAP" fee.
Because we can only distinguish fees by their descriptions, we try to match the description to the format string of the fee type enums. We also only require individual fee matches when we are charging more than one type of fees, which makes the change compatible with most existing use cases where only one fees is charged and the description field is ignored in the extension.
We expect the workflow to be that a registrar sends a domain check, and we reply with exactly what fees we are expecting, and then it will use the descriptions in the response to send us a domain create with the correct fees.
Note that we aggregate fees within the same FeeType together. Normally there will only be one fee per type, but in case of custom logic there could be more than one fee for the same type. There is no way to distinguish them as they both use the same description. So it is simpler to just aggregate them.
This CL also includes some reformatting that conforms to google-java-format output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=186530316
The START_DATE_SUNRISE phase allows registration of domains only with a signed mark. In all other respects - it is identical to the GENERAL_AVAILABILITY phase.
Note that Anchor Tenants bypass all checks, and are hence able to register domains without a signed mark.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=185534793
When enabled for a registrar, all EPP operations on premium domains that have
costs (e.g. creates, renews, transfers) will fail unless the EPP fee extension
is used to explicitly ack the amount of fee as part of the EPP transaction.
This ack is required regardless of whether premium fee acking is required at
the registry level. No data migration is necessary since false is the desired
default for this new attribute.
This CL also contains some slight refactoring of static utility methods used to
perform fee verification; there was short-circuiting at call-sites in two
places when what was really needed was two methods, one implementing additional
functionality on top of the other, and calling the inner method in the places
where short-circuiting had previously been necessary.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=184229363
This uses an extensibility mechanism similar to that of WhoisCommandFactory
and CustomLogicFactory, namely, that a fully qualified Java class is
specified in the YAML file for each environment with the allocation token
custom logic to be used. By default, this points to a no-op base class
that does nothing. Users that wish to add their own allocation token
custom logic can simply create a new class that extends
AllocationTokenCustomLogic and then configure it in their .yaml config
files.
This also renames the existing *FlowCustomLogic *Flow instance variables
from customLogic to flowCustomLogic, to avoid the potential confusion with
the new AllocationTokenCustomLogic class that also now exists.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=183003112
The next step is to add them for domain checks as well (which is simpler
because it doesn't involve validation).
This requires the addition of a TrimWhitespaceAdapter for XML JAXB objects,
which will prove useful for other @XmlValue attributes in the future.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181526726
The extension itself isn't used for anything yet; allocation tokens can be
passed for domain creates and checks but are ignored if present. This will
be changed in a subsequent CL that adds AllocationToken entities and related
logic. Usage of this extension in any other EPP flow will throw an
UnsupportedExtensionException.
The relevant spec is https://tools.ietf.org/html/draft-gould-allocation-token-04
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181343188
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238852
The assertThrows/expectThrows refactoring script does not use method
references, apparently.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089048
These testing helper functions can't be handled by the automatic refactoring
tool because they're taking in expected exception details as parameters.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178832406
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.
Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
The scheme is:
- loadBytes: returns a ByteSource of the data
- loadFile: returns a string using UTF8 encoding, optionally applying
substitutions
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177606406
Failing to use the fee extension during EAP can result in charges to registrars
that are radically different than what they may have been expecting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177597883
This makes the units explicit, which prevents confusion and bugs.
More information: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172455602
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Specifically, this prevents suspended registrars from creating domains or applications. Pending registrars already can't perform these actions because they get an error message when attempting to log in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170481338
The probers make a constant stream of create and delete calls, which we don't
want to account for when constructing transaction reports. This change will
cause only real TLDs to log create and delete transaction records.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166737801
After working further with domain deletes, I realized we'll need to record multiple reportingTimes under a single historyEntry when issuing a -1 counter to cancel grace-period adds. Since the TLD would be the only shared component within a record, we'll just duplicate it across all records to save an unnecessary layer of hierarchy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166261413
This is the second of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
Adds and renews each result in a +1 counter for the NET_ADDS/RENEWS_#_YR field,
which I've added simple (# of years, add or renew) -> Enum functions to get.
Allocates are just a special case of adds, and are counted in a similar manner.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165963249
Added validation on domain creation, preventing a domain from being created if
it equals an existing TLD. Added domain create tests for domains using
multi-part TLDs that shared suffixes and prefixes. Added host create tests for
hosts using multi-part TLDs that shared suffixes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164297749
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).
This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
Memcache is already off but now it's not in the code anymore.
This includes removing domain creation failfast, since that is actually
slower now than just running the flow - all you gain is a non-transactional
read over a transactional read, but the cost is that you always pay that
read, which is going to drive up latency.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158183506
We've determined that getting correctness semantics right, even
in the few cases that it is possible to do so (see linked bug for
audit) is not worth the bother in terms of highly complicated code
and potential bugs. This CL turns off memcache at the Ofy level
but doesn't rip out the annotations etc. so that we can quickly
turn it back on if this turns out to have been a mistake.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155227761
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
fail. We aren't ready to do this in prod yet, which is why the tests are still
marked @Ignore. The new tests fail if you change line 134 in Ofy to not use memcache
and either use the unchanged original DomainCreateFlow code, or use the new
inlined code and change loadWithMemcache() to load(). They pass with the new
inlined code that calls loadWithMemcache(), as long as the @Cache is added to
DomainResource.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154224748
As part of b/36599833, this makes FlowReporter log the tld(s) of every domain
flow it executes, so we can provide ICANN reporting totals on a per-TLD basis.
It also adds several other fields that we're computing anyway and which seem
useful, particularly for debugging any issues we see in production with the data
that we're attempting to record for ICANN reporting. The full set of fields is:
- commandType (e.g. "create", "info", "transfer")
- resourceType* (e.g. "domain", "contact", "host")
- flowClassName (e.g. "ContactCreateFlow", "DomainRestoreRequestFlow")
- targetId* (e.g. "ns1.foo.com", "bar.org", "contact-1234")
- targetIds* - plural of the above, for multi-resource checks
- tld** (e.g. "com", "co.uk") - extracted from targetId, lowercased
- tlds** - plural of the above, deduplicated, for multi-resource checks
* = only non-empty for resource flows (not e.g. login, logout, poll)
** = only non-empty for domain flows
Note that TLD extraction is deliberately very lenient to avoid the complexity
overhead of double-validation of the domain names in the common case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154070794
This fixes recording of number of attempts and command name on EPP
flows, which was broken because a separate metric builder was
being injected in two places, EppController and FlowRunner, with the
one injected into FlowRunner being discarded rather than having changes
applied to the same instance as in EppController.
This also adds a test that the metric is created successfully inside
a flow. Note that tests already exist for EppController to ensure that
the metric is recorded correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152306596
When a TLD is domain create restricted, every domain that is created under it will have both SERVER_TRANSFER_PROHIBITED and SERVER_UPDATE_PROHIBITED status applied on it. This way after a domain is created no registrar can change any settings on it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152266535
As part of b/36599833, this makes FlowRunner log the appropriate ICANN activity
report field name for each flow it runs as part of a structured JSON log
statement which can be parsed to generate ICANN activity reports (under the key
"icannActivityReportField").
In order to support this, we introduce an annotation for Flow classes called
@ReportingSpec and a corresponding enum of values for this annotation, which is
IcannReportingTypes.ActivityReportField, that stores the mapping of constant
enum values to field names.
The mapping from flows to fields is fairly obvious, with three exceptions:
- Application flows are all accounted under domains, since applications are
technically just deferred domain creates within the EPP protocol
- ClaimsCheckFlow is counted as a domain check
- DomainAllocateFlow is counted as a domain create
In addition, I've added tests to all the corresponding flows that we are
indeed logging what we expect.
We'll also need to log the TLD for this to be useful, but I'm doing that in a
follow-up CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151283411
During domain create/applicationcreate/allocate, domains that are on the reserved list(s) with nameserver restricted reservation type must set nameservers that are part of the allowed nameservers for that domain in the reserved list(s) applied to that TLD.
Additionally a boolean is added to Registry to indicate if a TLD is restricting domain create. If it is, only domains that are nameserver restricted can be registered.
For consistency with a similar feature that validates a TLD-wide nameserver whitelist, the per-domain nameserver validation is performed even when the operation is in super-user mode. Similarly, if a domain is nameserver restricted, nameservers must be supplied (i. e. the nameservers set cannot be empty) when registering the domain.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150641269
This also bypasses signed mark validation during domain creation if
the flow is being executed as superuser.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145435268
