TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
fail. We aren't ready to do this in prod yet, which is why the tests have a TODO
in them. The new tests fail if you change line 134 in Ofy to not use memcache
and either use the unchanged original flow code, or use the new
inlined code and change loadWithMemcache() to load(). They pass with the new
inlined code that calls loadWithMemcache(), as long as the @Cache is added to
DomainResource.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154457655
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
fail. We aren't ready to do this in prod yet, which is why the tests are still
marked @Ignore. The new tests fail if you change line 134 in Ofy to not use memcache
and either use the unchanged original DomainCreateFlow code, or use the new
inlined code and change loadWithMemcache() to load(). They pass with the new
inlined code that calls loadWithMemcache(), as long as the @Cache is added to
DomainResource.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154224748
No tests break because of this, but some tests that were previously
intentionally clearing memcache (for example, the test in
DomainInfoFlow that counts datastore calls) are now doing something
that is not a no-op anymore.
Also fix an incorrect comment about memcache in DatastoreHelper.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154087182
As part of b/36599833, this makes FlowReporter log the tld(s) of every domain
flow it executes, so we can provide ICANN reporting totals on a per-TLD basis.
It also adds several other fields that we're computing anyway and which seem
useful, particularly for debugging any issues we see in production with the data
that we're attempting to record for ICANN reporting. The full set of fields is:
- commandType (e.g. "create", "info", "transfer")
- resourceType* (e.g. "domain", "contact", "host")
- flowClassName (e.g. "ContactCreateFlow", "DomainRestoreRequestFlow")
- targetId* (e.g. "ns1.foo.com", "bar.org", "contact-1234")
- targetIds* - plural of the above, for multi-resource checks
- tld** (e.g. "com", "co.uk") - extracted from targetId, lowercased
- tlds** - plural of the above, deduplicated, for multi-resource checks
* = only non-empty for resource flows (not e.g. login, logout, poll)
** = only non-empty for domain flows
Note that TLD extraction is deliberately very lenient to avoid the complexity
overhead of double-validation of the domain names in the common case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154070794
This wasn't being recorded correctly because the clientId is only
set in LoginFlow after the flow succeeds, whereas we were
previously logging the clientId before executing the flow. This
adds special handling for LoginFlow. Note that we only set the
metric label to the clientId for valid registrar logins, to ensure
that metric cardinality doesn't grow unbounded (as it might if we
used every arbitrary string passed in as an attempted login).
This also refactors creation and handling of FakesAndMocksModule so
as to be able to make test assertions about EPP metrics from
integration flow tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154048280
Since this feeds into ICANN reporting, we don't want to muddy the data
there with dry-runs, which are always internal-only artifacts of tool usage
and shouldn't really count as real attempts to do SRS actions, since they
are always going to abort with no effect.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153856915
This primarily adds accessors to EppInput that will be used for flow reporting
logging in FlowReporter. Specifically, it adds:
- Optional<String> getResourceType() -> domain/host/contact
- Optional<String> getSingleTargetId() -> for SingleResourceCommands
And in addition, it adjusts getCommandName() so that it's now named
getCommandType() for better parallelism with the new getResourceType() (since
getResourceName() would be misleading), and it changes the value returned to be
lowercased, again for consistency. This isn't an issue because getCommandName()
isn't actually used anywhere right now (it was formerly used for EPP whitebox
metrics, but no longer due to recent changes there).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153851957
This prevents a possible failure mode of the logging where the logged
EPP input XML is very large (which can happen e.g. for domain creates
with large SMD values). In those cases, the XML might cause the overall
JSON string to be too large to fit within a single log entry [1], in which
case it gets split over multiple lines and breaks automatic parsing.
This mitigates that case by logging the EPP input (raw and base64-encoded)
in a separate log statement so that the more compact metadata (like clientId)
and derived values (like ICANN reporting field) will still be in an intact
JSON string even in that case, and can still be readily parsed. It's okay
if the actual EPP XML is harder to parse, since once we're logging the right
metadata fields we shouldn't need to automatically parse the EPP XML in any
normal cases.
[1] I haven't found this exact limit or splitting algorithm, or whether it's
a property of java logging or GAE log ingestion. The GAE logs page does note
that a single application log entry (within a request, which can have up to
1000 such entries) maxes out at 8KB, so that might be it:
https://cloud.google.com/appengine/docs/standard/java/logs/#writing_application_logs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153771335
Since this reporting is getting more complicated (see b/36599833), it'll
be better to have a dedicated class to encapsulate it, which also lets us
keep the tests separate and focus FlowRunner more on its core purpose of
actually running the flow.
Note that this doesn't move the legacy log statement logging because that
specifically must be logged from the FlowRunner.run() method to preserve
the existing log signature matching in our ICANN activity reporting query.
(The new statement is designed to be robust to moves like this since it
doesn't use the logging callsite to match log lines, and it's not in use
yet anyway.)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153762008
We now send PendingActionNotificationResponses in our poll messages upon completion of an asynchronous contact or host deletion. This is part 1 of 2, which begins logging Trid in all enqueued Host/Contact deletion flows for use in batch deletions, and optionally consuming the resultant Trid info to emit a Host/ContactPendingActionNotifcationResponse.
Part 2 will make this response emission non-optional, which will happen once the queue is cleared of all non-Trid containing tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153084197
For TLDs with domain create restriction. SERVER_TRANSFER_PROHIBITED and SERVER_UPDATE_PROHIBITED status codes
are automatically applied to newly created domains to make them immutable. When there is a legitimate for an update on a domain, the registry must first run nomulus update_server_locks to remove status before the registrar can request an update via EPP.
To eliminate the risk of the registry forgetting to reapply the codes after a update, we automatically re-apply these codes after a success update.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152533379
This fixes recording of number of attempts and command name on EPP
flows, which was broken because a separate metric builder was
being injected in two places, EppController and FlowRunner, with the
one injected into FlowRunner being discarded rather than having changes
applied to the same instance as in EppController.
This also adds a test that the metric is created successfully inside
a flow. Note that tests already exist for EppController to ensure that
the metric is recorded correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152306596
When a TLD is domain create restricted, every domain that is created under it will have both SERVER_TRANSFER_PROHIBITED and SERVER_UPDATE_PROHIBITED status applied on it. This way after a domain is created no registrar can change any settings on it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152266535
Since domain create restriction only applies to closed TLDs, flows like domain application create and domain application update does not apply, as the TLD never goes through sunrise period. Removing checks for domain create restrictions in these flows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152260673
As part of b/36599833, this makes FlowRunner log the appropriate ICANN activity
report field name for each flow it runs as part of a structured JSON log
statement which can be parsed to generate ICANN activity reports (under the key
"icannActivityReportField").
In order to support this, we introduce an annotation for Flow classes called
@ReportingSpec and a corresponding enum of values for this annotation, which is
IcannReportingTypes.ActivityReportField, that stores the mapping of constant
enum values to field names.
The mapping from flows to fields is fairly obvious, with three exceptions:
- Application flows are all accounted under domains, since applications are
technically just deferred domain creates within the EPP protocol
- ClaimsCheckFlow is counted as a domain check
- DomainAllocateFlow is counted as a domain create
In addition, I've added tests to all the corresponding flows that we are
indeed logging what we expect.
We'll also need to log the TLD for this to be useful, but I'm doing that in a
follow-up CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151283411
It was previously only using the name of the inner command XML element,
e.g. "Create", "Delete", "Update", etc. This wasn't very useful because
there was no way to discriminate between operations on different types
of EPP resources.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151131491
When updating domains, make sure that if the domains are nameserver restricted, the updated nameservers set on the domains are still consistent with the restriction.
When updating domains of a domain created restricted TLD, validate if the domain is still on the reserved list with nameserver restricted reservation. If it is not, there's likely some conflicting states of the domain that needs to be reconciled (e. g.the domain is removed from the reserved list after being created). Throws an exception in this case.
Also added missing tests for TLDs with nameserver whitelist.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150781935
During domain create/applicationcreate/allocate, domains that are on the reserved list(s) with nameserver restricted reservation type must set nameservers that are part of the allowed nameservers for that domain in the reserved list(s) applied to that TLD.
Additionally a boolean is added to Registry to indicate if a TLD is restricting domain create. If it is, only domains that are nameserver restricted can be registered.
For consistency with a similar feature that validates a TLD-wide nameserver whitelist, the per-domain nameserver validation is performed even when the operation is in super-user mode. Similarly, if a domain is nameserver restricted, nameservers must be supplied (i. e. the nameservers set cannot be empty) when registering the domain.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150641269
Now that transfers are always restricted to 1 year, it's unnecessary to store
extendedRegistrationYears on TransferData - it will always be equal to 1. This
simplifies logic in a few other places, e.g. RdeDomainImportAction.
I verified in BigQuery that no DomainBases exist with extendedRegistrationYears
values that aren't either null or equal to 1. At some point we should remove
the persisted fields from datastore via e.g. resaving all those domains, but
it's low priority and can wait until we have some more pressing migration.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150373897
This CL restricts domain transfer pricing lookups (on domain check and info) to
only support a 1-year period for inquiring about transfer fees. That treatment
matches what we do for domain restores, which are also always one year. This is
a followup to [] which disallowed actual transfer request flows from
specifying multi-year periods.
Since it's no longer necessary, this CL also changes the domain transfer pricing
logic to drop the years parameter, including removing the parameter from the
custom pricing logic TransferPriceParameters object.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150367839
EPP host spec section 3.2.5 requires that attempts to update hosts that are linked to different registrars must fail with error code 2305. This is complicated to do, as linked status is eventually consistent, and even more painful when checking links to those of different registrars.
This change forbids external-to-anything renames entirely.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150336754
This tweaks the logic that prohibits domain renews during pending transfers to just use the regular verifyNoDisallowedStatuses() check instead of a special check on TransferData with a custom exception. This is simpler and produces a better error message: we get "Operation disallowed by status: pendingTransfer" instead of "Object with given ID (foo.com) already has a pending transfer" (which is intended for use when denying a transfer request for an object already being transferred, not for this case).
For the record, we originally prohibited renews for domains in pending transfer because there's no good reason to do such a renew: b/12533793. But in fact our transfer server-approve logic relies heavily on this behavior, because otherwise the domain's expiration time computed in cloneProjectedAtTime() will reflect the transfer year added to the post-renew expiration time, whereas all the transfer server approve entities (e.g. new autorenew billing event) will reflect the pre-renew expiration time at the moment the transfer was requested. As such, it would be quite difficult to ever support a renew during pending transfer, since it would need to change many fields within the transfer server approve entities.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150325501
This changes ResourceStatusProhibitsOperationException so that we print out the list of StatusValues using their XML names rather than the literal enum name, i.e. we use "pendingDelete" rather than "PENDING_DELETE".
This seems more correct given that EPP clients will be used to seeing the status values in the XML representation, and it also matches the existing ResourceHasClientUpdateProhibitedException that hardcodes "clientUpdateProhibited":
http://[]/third_party/java_src/gtld/java/google/registry/flows/exceptions/ResourceHasClientUpdateProhibitedException.java?l=22&rcl=146111211
Also reorganized related test methods and added some missing tests, including for ContactTransferRequestFlow which previously had none. I also renamed the "clientProhibitedStatusValue" tests to instead say "statusValueNotClientSettable" to be clearer about what's being tested, and that it's not related to the "clientXXProhibited" statuses.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150248562
It turns out that this ICANN policy appears to prohibit transfers with
registration extensions other than 1 year (section A.8):
https://www.icann.org/resources/pages/policy-2012-03-07-en
This is backed up by the practical fact that we've never seen a registrar
request a transfer for any period other than one year.
And removing the support for multi-year transfers vastly simplifies
transfer logic and eliminates a bunch of annoying corner cases. Users
still can achieve the same thing by doing a 1-year transfer plus a
manual renewal afterwards for the remainder of the desired extension.
This change leaves in place lots of infrastructure to support multi-year
transfers that is now obsolete (e.g. TransferData.extendedRegistrationYears).
This should all be cleaned up, but it's a lower priority than fixing the
gap itself and insulating ourselves against needing to handle any real
multi-year transfer case. Once this CL goes in, we can start ignoring
extendedRegistrationYears entirely because it'll always be 1 year, which
makes the cleanup process easier.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150212864
Instead of only returning the most severe one, return all applicable ones. This is because the reserved list has grown to a list of types that are not strictly comparable but orthogonal to each other. We can no longer depend on the fact that the most severe type incorporates all properties of those beneath it. Therefore returning all of them and treat them one by one in the calling site is the correct behavior.
Due to constraint imposed in eppcom.xsd, during domain checks the response can only contain a reservation reason of fewer than 32 characters, therefore we are returning the message for the type with highest severity, in case of multiple reservation types for a label.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149776106
In fact, completely eviscerate cloneProjectedAtTime (to be removed in
a followup CL) in favor of doing the projection of transfers and the
loading of values from the superordinate domain at call sites. This
is one of the issues that blocked the memcache audit work, since the
load inside of cloneProjectedAtTime could not be controlled by the
caller.
Note: fixed a minor bug where a subordinate host created after its superordinate domain was last transferred should have lastTransferTime==null but was previously reporting the domain's lastTransferTime.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149769125
This fixes longstanding bug b/19430703 in which domain transfers that were
server-approved would only handle the autorenew grace period correctly if
the autorenew grace period was going to start within the transfer window.
If the autorenew grace period was already active (e.g. the domain had
recently autorenewed, before the transfer was requested), the logic would
miss it, even if it was going to be active throughout the transfer window
(i.e. it would still be active at the server-approval time).
When the autorenew grace period is active at the time a transfer is approved
(whether by the server or explicitly via DomainTransferApproveFlow), the
correct behavior is to essentially "cancel" the autorenew - the losing registrar
receives a refund for the autorenew charge, and the gaining registrar's transfer
extended registration years are applied to the expiration time as it was prior
to that autorenew. The way we implement this is that we just have the transfer
essentially "subsume" the autorenew - we deduct 1 year from the transfer's
extended registration years before extending the registration period from what
the expiration time is post-autorenew at the moment of transfer approval.
See b/19430703#comment17 for details on the policy justification; the only real
ICANN document about this is https://www.icann.org/news/advisory-2002-06-06-en,
but registrars informally document in many places that transfers will trigger
autorenew grace, e.g. see https://support.google.com/domains/answer/3251236
There are still a few parts of this bug that remain unfixed:
1) RdeDomainImportAction repeats a lot of logic when handling imported domains
that are in pending transfer, so it will also need to address this case in
some way, but the policy choices there are unclear so I'm waiting until we
know more about RDE import goals to figure out how to fix that.
2) Behavior at the millisecond edge cases is inconsistent - specifically, for
the case where a transfer is requested such that the automatic transfer
time is exactly the domain's expiration time (down to the millisecond),
the correct behavior is a little unclear and this CL for now ignores this
issue in favor of getting a fix for 99.999% of the issue into prod. See
newly created b/35881941 for the gory details.
Also, there are parts of this bug that will be fixed as parts of either
b/25084229 (transfer exDate computations) or b/35110537 (disallowing transfers
with extended registration years other than 1), both of which are less pressing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149024269
This documents some slightly spooky behavior around domains that have an expiration time within their pendingDelete window (meaning the whole period from DomainDeleteFlow running to the actual deletionTime, not just the 5-day pendingDelete grace period). They will experience an autorenew in terms of expiration time and grace period status due to cloneProjectedAtTime(), but without the usual artifacts of an autorenew (billing event and poll message).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149019980
All domain/host names should be stored in their canonical forms (puny-
coded and lower-cased). This validation is already in the flows, but
this adds protection against bad data from other sources, e.g. admin
consoles or RDE imports.
This also removes an old work-around that temporarily suspended this
validation for superusers, because we used to have non-canonicalized
data in the system. The non-canonicalized data has since all been
cleaned up, so this work-around is no longer necessary.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146799558
This is a cleanup in preparation for the next change that does a lot
of work with subordinate hosts, to make it easier to reason about in
complex code.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146689904
There are still some options in RegistryConfig that can't be configured
in YAML, but it's not clear why anyone would need to change them from
their default values.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146482435
This bug is about a bad use of Optional.
We were checking == null instead of .isPresent(), so the check
always passed, and we always set a lastSubordinateTime when
updating hosts, even if the host was external and should have
had a null value in that field.
There is almost certainly bad data in prod in the sense
that any external host that was ever updated will have a value
for this field instead of null. However, this is not
consequential as the field is entirely meaningless for
external hosts, and will be properly reset if the host is
ever moved to be internal.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146363178
* Remove LINKED when loading an EppResource
* Enforce that you can't add it to a resource
* Ignore LINKED on xjc import of contacts and hosts
After running ResaveAllEppResourcesAction we will no
longer have persisted LINKED statuses in datastore.
In the process of writing this I discovered that RDAP
treats LINKED like any other status value and returns
the persisted value rather than the derived one. Since
this is an existing bug and is orthogonal to the changes
in this CL, I am addressing it in a separate CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145585227
This also bypasses signed mark validation during domain creation if
the flow is being executed as superuser.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145435268
