We ran into a bunch of prober deployment issues this past week when
attempting to spin up a new cluster because the newly created prober
TLDs had null values for the dnsWriter field. Given that VoidDnsWriter
exists, we can require that dnsWriter always be set, and have people
use that if DNS publishing is not required.
Also cleans up a bunch of related inconsistent exception messages and
tests not verifying said exception messages properly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154325830
TESTED=The test fails if you change line 134 in Ofy to not use memcache
and use the unchanged original Registry.get() code. This is the
expected behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154226534
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
fail. We aren't ready to do this in prod yet, which is why the tests are still
marked @Ignore. The new tests fail if you change line 134 in Ofy to not use memcache
and either use the unchanged original DomainCreateFlow code, or use the new
inlined code and change loadWithMemcache() to load(). They pass with the new
inlined code that calls loadWithMemcache(), as long as the @Cache is added to
DomainResource.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154224748
This avoids loading all contacts and hosts before the failfast runs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154179321
The requirement will instead be enforced on registrar contact that is visible in WHOIS as abuse contact.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154102111
As part of b/36599833, this makes FlowReporter log the tld(s) of every domain
flow it executes, so we can provide ICANN reporting totals on a per-TLD basis.
It also adds several other fields that we're computing anyway and which seem
useful, particularly for debugging any issues we see in production with the data
that we're attempting to record for ICANN reporting. The full set of fields is:
- commandType (e.g. "create", "info", "transfer")
- resourceType* (e.g. "domain", "contact", "host")
- flowClassName (e.g. "ContactCreateFlow", "DomainRestoreRequestFlow")
- targetId* (e.g. "ns1.foo.com", "bar.org", "contact-1234")
- targetIds* - plural of the above, for multi-resource checks
- tld** (e.g. "com", "co.uk") - extracted from targetId, lowercased
- tlds** - plural of the above, deduplicated, for multi-resource checks
* = only non-empty for resource flows (not e.g. login, logout, poll)
** = only non-empty for domain flows
Note that TLD extraction is deliberately very lenient to avoid the complexity
overhead of double-validation of the domain names in the common case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154070794
This wasn't being recorded correctly because the clientId is only
set in LoginFlow after the flow succeeds, whereas we were
previously logging the clientId before executing the flow. This
adds special handling for LoginFlow. Note that we only set the
metric label to the clientId for valid registrar logins, to ensure
that metric cardinality doesn't grow unbounded (as it might if we
used every arbitrary string passed in as an attempted login).
This also refactors creation and handling of FakesAndMocksModule so
as to be able to make test assertions about EPP metrics from
integration flow tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154048280
Since this feeds into ICANN reporting, we don't want to muddy the data
there with dry-runs, which are always internal-only artifacts of tool usage
and shouldn't really count as real attempts to do SRS actions, since they
are always going to abort with no effect.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153856915
Add build rules for creating "nocron" war and ear files for use during
database restores in production and alpha.
After a little consideration, I think this is the right way to do this. If we
want to set it up in other environments, we'll need to add these rules for
them, too, and at that point it may be worthwhile to wrap the rules we want in
a macro, but for now this is fairly clear and not execessively redundant.
This change works by modifying the zip_file rule to allow us to exclude
"cron.xml". I have verified that:
- The alpha "nocron" files are generated without cron.xml.
- The production "nocron" files are generated without cron.xml.
- I can do a full push to alpha based on deploy_nocron which doesn't install a
new cron.
- The normal production and alpha war-files _do_ contain cron.xml.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153853205
This primarily adds accessors to EppInput that will be used for flow reporting
logging in FlowReporter. Specifically, it adds:
- Optional<String> getResourceType() -> domain/host/contact
- Optional<String> getSingleTargetId() -> for SingleResourceCommands
And in addition, it adjusts getCommandName() so that it's now named
getCommandType() for better parallelism with the new getResourceType() (since
getResourceName() would be misleading), and it changes the value returned to be
lowercased, again for consistency. This isn't an issue because getCommandName()
isn't actually used anywhere right now (it was formerly used for EPP whitebox
metrics, but no longer due to recent changes there).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153851957
This prevents a possible failure mode of the logging where the logged
EPP input XML is very large (which can happen e.g. for domain creates
with large SMD values). In those cases, the XML might cause the overall
JSON string to be too large to fit within a single log entry [1], in which
case it gets split over multiple lines and breaks automatic parsing.
This mitigates that case by logging the EPP input (raw and base64-encoded)
in a separate log statement so that the more compact metadata (like clientId)
and derived values (like ICANN reporting field) will still be in an intact
JSON string even in that case, and can still be readily parsed. It's okay
if the actual EPP XML is harder to parse, since once we're logging the right
metadata fields we shouldn't need to automatically parse the EPP XML in any
normal cases.
[1] I haven't found this exact limit or splitting algorithm, or whether it's
a property of java logging or GAE log ingestion. The GAE logs page does note
that a single application log entry (within a request, which can have up to
1000 such entries) maxes out at 8KB, so that might be it:
https://cloud.google.com/appengine/docs/standard/java/logs/#writing_application_logs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153771335
Since this reporting is getting more complicated (see b/36599833), it'll
be better to have a dedicated class to encapsulate it, which also lets us
keep the tests separate and focus FlowRunner more on its core purpose of
actually running the flow.
Note that this doesn't move the legacy log statement logging because that
specifically must be logged from the FlowRunner.run() method to preserve
the existing log signature matching in our ICANN activity reporting query.
(The new statement is designed to be robust to moves like this since it
doesn't use the logging callsite to match log lines, and it's not in use
yet anyway.)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153762008
This is required by ICANN Consistent Labeling & Display policy that WHOIS domain query response contains registrar abuse contact's phone number and email address. Add a helper function to load registrar contact of a certain type for a given registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153606137
Make it clear that all the user need to do to rectify is to provide a phone number
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153191178
One is an unnecessary import and the other is an incorrectly named
Javadoc parameter.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153095269
We now send PendingActionNotificationResponses in our poll messages upon completion of an asynchronous contact or host deletion. This is part 1 of 2, which begins logging Trid in all enqueued Host/Contact deletion flows for use in batch deletions, and optionally consuming the resultant Trid info to emit a Host/ContactPendingActionNotifcationResponse.
Part 2 will make this response emission non-optional, which will happen once the queue is cleared of all non-Trid containing tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153084197
Leave allowedOauthClientIds empty instead of moving the placeholder client ids over.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152967043
Replace KeystoreKeyring with ComparatorKeyring between KeystoreKeyring and
KmsKeystore. In the opensource version, will replace DummyKeyring with
KmsKeyring directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152893767
Previously, GenerateEscrowDepositCommand generated the deposit itself. Channeling it through the existing deposit generation code make things more maintainable.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152847950
For TLDs with domain create restriction. SERVER_TRANSFER_PROHIBITED and SERVER_UPDATE_PROHIBITED status codes
are automatically applied to newly created domains to make them immutable. When there is a legitimate for an update on a domain, the registry must first run nomulus update_server_locks to remove status before the registrar can request an update via EPP.
To eliminate the risk of the registry forgetting to reapply the codes after a update, we automatically re-apply these codes after a success update.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152533379
RdeStagingAction always processed all RDE and BRDA deposits currently outstanding, updating the cursors appropriately and kicking off the upload job. Sometimes we don't want all that. We just want to create a specific deposit by hand, without modifying the cursors or uploading. This CL adds parameters to support that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152415959
This is better than calling assertTldExists() inside a for loop because you can throw a single exception reporting all bad TLDs at once rather than only getting as far as the first failure. And then it's also a one-liner instead of 3 lines.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152412876
The code to authenticate and authorize incoming requests (including via OAuth) has been in the system. This CL actually turns it on, since we are satisfied from logging information that it is not unjustly denying access.
Auth settings are also updated on a few commands missed earlier.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152381820
{css $foo} does not do anything. the correct syntax is
{css selector} (no dollar sign)
or
{css $prefix, selector}
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152375039
It has 5 parameters, which is pushing it for a static factory method, and we're anticipating adding more for work that Larry is doing. Using a builder is preferable here since it makes it harder to accidentally mis-order the parameters (since @AutoValue's generated constructor is sensitive to parameter ordering).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152328220
This fixes recording of number of attempts and command name on EPP
flows, which was broken because a separate metric builder was
being injected in two places, EppController and FlowRunner, with the
one injected into FlowRunner being discarded rather than having changes
applied to the same instance as in EppController.
This also adds a test that the metric is created successfully inside
a flow. Note that tests already exist for EppController to ensure that
the metric is recorded correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152306596
RegistrarContacts are allowed to have no "types" specified (they can also have multiple types). However, the registrar console displays the contacts for the logged-in registrar grouped by type into sections, and contacts with no type were simply being omitted from the listing, which was confusing because you can't even log into the console unless your email is listed as a contact, and yet you might then visit the contact settings page and see (apparently) no configured contacts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152302159
When a TLD is domain create restricted, every domain that is created under it will have both SERVER_TRANSFER_PROHIBITED and SERVER_UPDATE_PROHIBITED status applied on it. This way after a domain is created no registrar can change any settings on it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152266535
TldFanoutAction now returns an HTTP response detailing the actions it's taking.
The format is as follows:
OK: Launched the following 3 tasks in queue the-queue
- Task: task-a6ad250b-a9d8-427d-bbf7-eb736e6f4dcb, tld: com, endpoint: /the/servlet/com
- Task: task-cf6c4bb4-0542-411e-ae4d-723beec09e9c, tld: net, endpoint: /the/servlet/net
- Task: task-57899661-fc3f-4049-a265-d6604051406e, tld: org, endpoint: /the/servlet/org
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152264954
Since domain create restriction only applies to closed TLDs, flows like domain application create and domain application update does not apply, as the TLD never goes through sunrise period. Removing checks for domain create restrictions in these flows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152260673
We now attempt to retry Whois queries in the event of a short-lived error. Currently, we consider 'DatastoreTimeoutException' and 'DatastoreFailureException' as transient.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152044934
Cloned from CL 149476124 by 'g4 patch'.
Original change by shikhman@shikhman:registry-secrets-2:897:citc on 2017/03/07 15:37:09.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151950797
This adds a new view table that contains the registrar id, the currency-specific billing account id and the corresponding currency in latest_billing dataset based on latest_snapshot dataset.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151363209
Join RegistrarAccountId table to BillingData to append additional billingAccountId column
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151362466
As part of b/36599833, this makes FlowRunner log the appropriate ICANN activity
report field name for each flow it runs as part of a structured JSON log
statement which can be parsed to generate ICANN activity reports (under the key
"icannActivityReportField").
In order to support this, we introduce an annotation for Flow classes called
@ReportingSpec and a corresponding enum of values for this annotation, which is
IcannReportingTypes.ActivityReportField, that stores the mapping of constant
enum values to field names.
The mapping from flows to fields is fairly obvious, with three exceptions:
- Application flows are all accounted under domains, since applications are
technically just deferred domain creates within the EPP protocol
- ClaimsCheckFlow is counted as a domain check
- DomainAllocateFlow is counted as a domain create
In addition, I've added tests to all the corresponding flows that we are
indeed logging what we expect.
We'll also need to log the TLD for this to be useful, but I'm doing that in a
follow-up CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151283411
This is a follow-up to Lai's refactoring of the get reservation types
code to return a set rather than a single type. Since we're always
returning a set now, the more natural way to represent a label that is
not reserved is to return an empty set rather than a set containing
UNRESERVED.
Also fixes some minor style issues I ran across regarding static
importing and test method naming that I ran across (no logic
implications).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151132116
It was previously only using the name of the inner command XML element,
e.g. "Create", "Delete", "Update", etc. This wasn't very useful because
there was no way to discriminate between operations on different types
of EPP resources.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151131491
