Commit graph

719 commits

Author SHA1 Message Date
mmuller
d4f222cecd Fix RestoreCommitLogsCommandTest in kokoro
Convert RestoreCommitLogsCommandTest use of generics and mockito to a form
that works with the kokoro build:

- Replace ImmutableMap<String, Object> with ImmutableMap<String, ?>.
- Replace any() as a matcher for MediaType with an "eq()" matcher.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157148910
2017-06-05 18:17:09 -04:00
mmuller
b733443eab Create nomulus tool restore command
Create a nomulus tool restoreCommitLogs command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157103177
2017-06-05 18:17:09 -04:00
mountford
ddb0f5692e Add golden files for request component routing maps
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157026188
2017-06-05 18:17:09 -04:00
mcilwain
bb67841884 Add metrics for async batch operation processing
We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:

* An incrementable metric for each async task processed (split out by
  type of task and result).
* Two event metrics for processing time between when a task is enqueued
  and when it is processed -- tracked separately for contact/host
  deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
  run (this is usually 0). Tracked separately for contact/host deletion
  and DNS refresh on host rename.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
2017-06-05 18:17:09 -04:00
mcilwain
1adeb57fea Add a Fibonacci fitter for metrics bucketing
A Fibonacci fitter is useful in situations where you want more precision on the
low end than an ExponentialFitter with exponent base 2 provides without the
hassle of dealing with non-integer boundaries, such as would be created by an
exponential fitter with a base of less than 2. Fibonacci fitters are ideal for
integer metrics that are bounded across a certain range, e.g. integers between 1
and 1,000.

This also cleans up some unit test comments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156773367
2017-05-23 17:22:49 -04:00
nickfelt
9ee9229456 Fix message for SuperordinateDomainInPendingDeleteException
The StatusProhibitsOperationException superclass constructor just wants a raw message to return to the client, so we should pass it an explanation rather than just the superordinate domain name.  In fact, I think the superordinate domain name isn't really necessary since it should be obvious from the hostname in the create/update request what the superordinate domain would be.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156747185
2017-05-23 17:22:49 -04:00
mountford
fb21946366 Fix parameter problem in GenerateEscrowDepositCommand
It was setting the TLD and watermark date parameters as comma-separated lists. What it really needs to do is set the parameters multiple times, once for each value.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156741051
2017-05-23 17:22:49 -04:00
nickfelt
65aaeccfc6 Fix TaskQueueHelper param matching for pull queue tasks
This fixes TaskQueueHelper methods and MatchableTaskInfo so that the .param() matching works for pull queues, by parsing the payload for URL-encoded parameters more liberally.  As such, it updates all the places where formerly we were hacking around this by manually constructing the expected payloads and using TaskMatcher.payload() instead.

It also adds a TaskQueueHelper.assertTasksEnqueued() overload that accepts an Iterable<TaskStateInfo> so that you can cleanly assert that a queue contains the same tasks that were returned via a previous call to getQueueInfo("queue").getTaskInfo().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156604901
2017-05-23 17:22:49 -04:00
jianglai
3f5735ed4b Add Carriage Return in WHOIS disclaimer text
WHOIS disclaimer is read from the config yaml file directly and the line breaks
within the text does not contain carriage return, which ICANN requires. This
CL fixes the non-compliance.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156359526
2017-05-23 17:22:49 -04:00
cgoldfeder
c9d7e75946 Cache Registrars in memory
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.

While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
2017-05-17 12:26:11 -04:00
mountford
74a0defef3 Add the ability to generate RDE deposits in lenient mode
We will not want to run this under normal circumstances, but in cases (such as PDT testing in sandbox) where it's desirable to generate an escrow deposit even if it isn't technically valid XML, this gives us that option. Manual-mode RDE generation is also changed so that, if no watermark date is specified, it defaults to the previous midnight, to better support running of RDE in sandbox to catch data problems.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156108295
2017-05-17 12:22:49 -04:00
jianglai
35624ace08 Require currency specific billingAccountId for registrar
For the currency specific billing account ids to work properly, any REAL registrar that is allowed to register domains in a certain TLD must have a billing account id for the currency of that TLD.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864711
2017-05-17 12:09:46 -04:00
jianglai
6e6a340113 Allow multiple domain creation in nomulus tool
This helps creating test domains. Also fixed a bunch of bad test cases.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864065
2017-05-17 12:08:10 -04:00
mcilwain
ef1487cb57 Make name and address fields required on Registrar
The absence of these fields causes RDE failures, so they are in effect
required on any functioning registry system. We are currently
experiencing problems in sandbox caused by null values on these fields.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155474895
2017-05-17 11:44:46 -04:00
jianglai
8892656722 Make WHOIS domain query return conform to ICANN CL&D policy
ICANN's policy can be found here:
https://www.icann.org/resources/pages/rdds-labeling-policy-2017-02-01-en

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155375811
2017-05-17 11:38:28 -04:00
jianglai
2846f9c6b9 This CL include changes in the registrar console that makes it possible to designate an abuse contact in domain WHOIS record, per ICANN's CL&D requirement.
Frontend validation: ensures that only one WHOIS abuse contact exist per registrar. Any existing WHOIS abuse contact will be overridden when a new one is designated.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155289097
2017-05-17 11:36:53 -04:00
cgoldfeder
275d6ddc10 Disable memcache completely
We've determined that getting correctness semantics right, even
in the few cases that it is possible to do so (see linked bug for
audit) is not worth the bother in terms of highly complicated code
and potential bugs. This CL turns off memcache at the Ofy level
but doesn't rip out the annotations etc. so that we can quickly
turn it back on if this turns out to have been a mistake.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155227761
2017-05-17 11:35:19 -04:00
cgoldfeder
608e121a37 Set the premium cache duration in tests to 60s not 60ms
This was causing test flakiness.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155225717
2017-05-17 11:33:44 -04:00
mmuller
3eef5095d4 Deal with gaps and forks in commit history
If there are gaps or forks in the commit log history files, raise an exception
and display the possible sets of files in the log files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155219410
2017-05-17 11:30:36 -04:00
jianglai
7a3852b5a5 Add visibleInWhoisAsAbuse field in RegistrarContact
Also added corresponding getters and setters for the new field. Note that
nothing has changed on the RDAP front for now, as the CL&D only concerns WHOIS.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155116134
2017-05-17 11:27:24 -04:00
guyben
93c2a1e4f0 Make JSch injection lazy
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154958750
2017-05-17 11:22:53 -04:00
guyben
5e7834b00e Wrap Kms decrypt with Retrier
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154861871
2017-05-03 11:10:53 -04:00
mountford
8bb66dd365 Add delete_host nomulus tool command
This is needed to delete an invalid host resulting from a previous bug.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154757425
2017-05-03 11:08:02 -04:00
mcilwain
e6249196af Fix formatting of DeleteDomainCommand
It was pretty off for some reason, so I just ran
google-java-format on it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154756158
2017-05-03 11:06:19 -04:00
nickfelt
f640d765e8 Log EppExceptions in EppController at INFO (vs FlowRunner at WARNING)
The logging for exceptions in FlowRunner - always at WARNING - has long been sub-optimal.  For EppExceptions it's too aggressive/spammy to log at WARNING because it's generally not actionable - EppException gets properly thrown for all kinds of ordinary reasons (trying to create a resource when one already exists with that foreign key) and/or for client misbehavior that we can't control (sending bad parameter values, etc.).  For non-EppException RuntimeExceptions, it's redundant with existing logging in EppController.

This CL resolves this by removing that logging in FlowRunner entirely in favor of the EppController logging, where we're now logging EppExceptions at INFO in parallel with the existing logging of RuntimeExceptions at SEVERE.  This has the benefit that we're now logging EppExceptions that come from FlowPicker (by way of EppExceptionInProviderException),  which previously were unlogged.

Note however that this does mean that in places where we run FlowRunner without EppController - exclusively test code as it stands today - we'd no longer be logging EppExceptions.  If that seems like a loss, we could either reinstate logging there (at INFO) and just deal with redundant messages for most EppExceptions, or we could add it manually to places where we call FlowRunner.run() in tests and avoid the redundancy that way.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154733365
2017-05-03 11:04:51 -04:00
mountford
11e7374c0f Fix bug which allowed creation of hosts with superordinate domains in pending delete state.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154716883
2017-05-03 11:03:23 -04:00
cgoldfeder
386d2bc6be Make Registrar.loadByClientId explicitly use memcache
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154712979
2017-05-03 11:01:54 -04:00
nickfelt
f9df4cf3f1 Revert unintentional no-op log handler attachment
This snuck in from [] but was just a vestige of when I was testing with the tests still in FlowRunner test and wanted to make sure they still passed with the logging moved to FlowReporter.  Now that the tests are in FlowReporterTest and FlowReporter is mocked out here, attaching the handler to that logger is a no-op.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154601472
2017-05-03 11:00:23 -04:00
mcilwain
876552f1ce Add more EPP metric lifecycle tests
This also refactors some of the existing EPP XML testdata files
to use a consistent wildcard naming scheme.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154569632
2017-05-03 10:58:51 -04:00
cgoldfeder
d4d02f8977 Add a test for Registrar.loadByClientId not being enrolled in a transaction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154470388
2017-05-03 10:52:46 -04:00
mcilwain
8025c7547b Add flow-level EPP metric tests
This also adds an assertion helper for EPP metrics.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154458134
2017-05-03 10:51:12 -04:00
cgoldfeder
f1129ea2b1 Make DomainInfoFlow (and application info) explicitly hit memcache
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
    fail. We aren't ready to do this in prod yet, which is why the tests have a TODO
    in them. The new tests fail if you change line 134 in Ofy to not use memcache
    and either use the unchanged original flow code, or use the new
    inlined code and change loadWithMemcache() to load(). They pass with the new
    inlined code that calls loadWithMemcache(), as long as the @Cache is added to
    DomainResource.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154457655
2017-05-03 10:49:38 -04:00
cgoldfeder
927eb43cbc Un-Ignore a test in DomainApplicationCreateFlowTest
A little injection-foo makes this test possible to run.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154442134
2017-05-03 10:44:56 -04:00
nickfelt
40fa9ff022 Fix some direct uses of ForeignKey(C/H/D)Index
Mostly these are calls to ForeignKeyIndex.create() (a static method) via subclasses, which is pretty misleading in this case since the type of the return value has nothing to do with the subclass you're qualifying the static method call with (the returned type depends only on the type of the EppResource parameter).

Note however though that while the style guide indeed prohibits qualifying static member references with things other than the class name, the subclassing case is apparently not considered subject to that prohibition in general:
https://groups.google.com[]d/msg/java-style/8ViX-Rh2_sc/48n2lz5nAAAJ

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154331605
2017-05-03 10:40:21 -04:00
mcilwain
d30f9411d8 Require that DNS writer be set on Registry entities
We ran into a bunch of prober deployment issues this past week when
attempting to spin up a new cluster because the newly created prober
TLDs had null values for the dnsWriter field. Given that VoidDnsWriter
exists, we can require that dnsWriter always be set, and have people
use that if DNS publishing is not required.

Also cleans up a bunch of related inconsistent exception messages and
tests not verifying said exception messages properly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154325830
2017-05-03 10:38:49 -04:00
cgoldfeder
d7da112c19 Put back accidentally deleted assertion
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154300634
2017-05-03 10:35:44 -04:00
cgoldfeder
3f9501e469 Remove obsolete comment.
init() is from pre-flattening days

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154256803
2017-04-26 11:21:35 -04:00
mountford
e00b0440b2 Roll back accidental inclusion of non-public Mockito helper methods
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154232020
2017-04-26 11:19:59 -04:00
cgoldfeder
4eba2ea07a Make Registry.get(tld) hit memcache explicitly.
TESTED=The test fails if you change line 134 in Ofy to not use memcache
    and use the unchanged original Registry.get() code. This is the
    expected behavior.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154226534
2017-04-26 11:18:02 -04:00
cgoldfeder
9e61f1d6ef Make failfastForCreate for domain and application creates explicitly hit memcache
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
    fail. We aren't ready to do this in prod yet, which is why the tests are still
    marked @Ignore. The new tests fail if you change line 134 in Ofy to not use memcache
    and either use the unchanged original DomainCreateFlow code, or use the new
    inlined code and change loadWithMemcache() to load(). They pass with the new
    inlined code that calls loadWithMemcache(), as long as the @Cache is added to
    DomainResource.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154224748
2017-04-26 11:16:33 -04:00
mountford
cb145e0721 Remove some unnecessary uses of MockitoJUnitRunner, which is discouraged
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154202151
2017-04-26 11:15:07 -04:00
cgoldfeder
51cfc12c1d Add MemcacheHelper to simplify memcache usage in tests
And make two callsites use it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154189438
2017-04-26 11:12:17 -04:00
jianglai
33fabfff04 Remove abuse contact requirement
The requirement will instead be enforced on registrar contact that is visible in WHOIS as abuse contact.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154102111
2017-04-26 11:07:48 -04:00
cgoldfeder
b6e50f0467 Turn on memcache in flow tests.
No tests break because of this, but some tests that were previously
intentionally clearing memcache (for example, the test in
DomainInfoFlow that counts datastore calls) are now doing something
that is not a no-op anymore.

Also fix an incorrect comment about memcache in DatastoreHelper.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154087182
2017-04-26 11:04:56 -04:00
mcilwain
b15d715dc0 Fix two more Java 7 missing generic type errors
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154086209
2017-04-26 11:03:29 -04:00
nickfelt
f296b225af Make FlowReporter log tld and various other fields
As part of b/36599833, this makes FlowReporter log the tld(s) of every domain
flow it executes, so we can provide ICANN reporting totals on a per-TLD basis.

It also adds several other fields that we're computing anyway and which seem
useful, particularly for debugging any issues we see in production with the data
that we're attempting to record for ICANN reporting.  The full set of fields is:

  - commandType (e.g. "create", "info", "transfer")
  - resourceType* (e.g. "domain", "contact", "host")
  - flowClassName (e.g. "ContactCreateFlow", "DomainRestoreRequestFlow")
  - targetId* (e.g. "ns1.foo.com", "bar.org", "contact-1234")
  - targetIds* - plural of the above, for multi-resource checks
  - tld** (e.g. "com", "co.uk") - extracted from targetId, lowercased
  - tlds** - plural of the above, deduplicated, for multi-resource checks

* = only non-empty for resource flows (not e.g. login, logout, poll)
** = only non-empty for domain flows

Note that TLD extraction is deliberately very lenient to avoid the complexity
overhead of double-validation of the domain names in the common case.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154070794
2017-04-26 10:59:09 -04:00
mcilwain
c1643fa3cd Correctly set clientId on EPP metrics in LoginFlow
This wasn't being recorded correctly because the clientId is only
set in LoginFlow after the flow succeeds, whereas we were
previously logging the clientId before executing the flow. This
adds special handling for LoginFlow. Note that we only set the
metric label to the clientId for valid registrar logins, to ensure
that metric cardinality doesn't grow unbounded (as it might if we
used every arbitrary string passed in as an attempted login).

This also refactors creation and handling of FakesAndMocksModule so
as to be able to make test assertions about EPP metrics from
integration flow tests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154048280
2017-04-26 10:56:13 -04:00
nickfelt
7dc3ddfc14 Skip FlowReporter logging for dry runs
Since this feeds into ICANN reporting, we don't want to muddy the data
there with dry-runs, which are always internal-only artifacts of tool usage
and shouldn't really count as real attempts to do SRS actions, since they
are always going to abort with no effect.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153856915
2017-04-26 10:54:43 -04:00
nickfelt
c5d6a1f6fb Add and adjust EppInput accessors for more useful properties
This primarily adds accessors to EppInput that will be used for flow reporting
logging in FlowReporter.  Specifically, it adds:

 - Optional<String> getResourceType() -> domain/host/contact
 - Optional<String> getSingleTargetId() -> for SingleResourceCommands

And in addition, it adjusts getCommandName() so that it's now named
getCommandType() for better parallelism with the new getResourceType() (since
getResourceName() would be misleading), and it changes the value returned to be
lowercased, again for consistency.  This isn't an issue because getCommandName()
isn't actually used anywhere right now (it was formerly used for EPP whitebox
metrics, but no longer due to recent changes there).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153851957
2017-04-26 10:51:47 -04:00
nickfelt
f2c6021db0 Split FlowReporter logging into two lines for robustness
This prevents a possible failure mode of the logging where the logged
EPP input XML is very large (which can happen e.g. for domain creates
with large SMD values).  In those cases, the XML might cause the overall
JSON string to be too large to fit within a single log entry [1], in which
case it gets split over multiple lines and breaks automatic parsing.

This mitigates that case by logging the EPP input (raw and base64-encoded)
in a separate log statement so that the more compact metadata (like clientId)
and derived values (like ICANN reporting field) will still be in an intact
JSON string even in that case, and can still be readily parsed.  It's okay
if the actual EPP XML is harder to parse, since once we're logging the right
metadata fields we shouldn't need to automatically parse the EPP XML in any
normal cases.

[1] I haven't found this exact limit or splitting algorithm, or whether it's
a property of java logging or GAE log ingestion.  The GAE logs page does note
that a single application log entry (within a request, which can have up to
1000 such entries) maxes out at 8KB, so that might be it:
https://cloud.google.com/appengine/docs/standard/java/logs/#writing_application_logs

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153771335
2017-04-26 10:50:13 -04:00