Commit graph

1175 commits

Author SHA1 Message Date
nickfelt
60022ae315 Remove outdated comment in RdeUploadActionTest
I think this comment was meant to be the justification for not using "localhost" (aka a hostname) in the URLs, because jsch would mangle it.  However, we already cut over to using "localhost" in [] to avoid a dependency on IPv4, and it's been fine.  So this comment no longer makes sense.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158063880
2017-06-06 10:57:56 -04:00
guyben
75bb1aacdc Update cloudkms_java with the non-beta version
Changed [] to use v1 instead of v1beta1, and replaced v1beta1 with v1 in all the java files.

If there is special build rules for open-source etc. that also need to be updated, or non "TAP-able" tests that need to be run, please check and see if they are OK.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157895888
2017-06-05 18:17:09 -04:00
mountford
3bbf356497 Fix bug in registrar contact nomulus command
The command was set up such that an update without any contact types specified would clear out the list, instead of leaving them unchanged, as it should.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157766429
2017-06-05 18:17:09 -04:00
mountford
6cc3ee5e15 Add option to registrar_contact command to modify the visible_in_domain_whois_as_abuse flag
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157588734
2017-06-05 18:17:09 -04:00
mountford
009522d078 Change auth settings on RdeStagingAction
[] opened up the settings on RdeStaging, in order to make it usable by the nomulus tool. But in retrospect, we think that all we needed to do was support the POST method, not loosen the auth settings, since nomulus invokes RdeStaging via a task queue. Removing the looser auth settings will bring this action into line with other backend actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157262595
2017-06-05 18:17:09 -04:00
mountford
bc63d80a5e Change auth settings on some tools service actions
This brings the affected actions into line with the settings on other similar actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157259842
2017-06-05 18:17:09 -04:00
mmuller
d4f222cecd Fix RestoreCommitLogsCommandTest in kokoro
Convert RestoreCommitLogsCommandTest use of generics and mockito to a form
that works with the kokoro build:

- Replace ImmutableMap<String, Object> with ImmutableMap<String, ?>.
- Replace any() as a matcher for MediaType with an "eq()" matcher.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157148910
2017-06-05 18:17:09 -04:00
mmuller
b733443eab Create nomulus tool restore command
Create a nomulus tool restoreCommitLogs command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157103177
2017-06-05 18:17:09 -04:00
mountford
ddb0f5692e Add golden files for request component routing maps
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157026188
2017-06-05 18:17:09 -04:00
mcilwain
bb67841884 Add metrics for async batch operation processing
We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:

* An incrementable metric for each async task processed (split out by
  type of task and result).
* Two event metrics for processing time between when a task is enqueued
  and when it is processed -- tracked separately for contact/host
  deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
  run (this is usually 0). Tracked separately for contact/host deletion
  and DNS refresh on host rename.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
2017-06-05 18:17:09 -04:00
mcilwain
1adeb57fea Add a Fibonacci fitter for metrics bucketing
A Fibonacci fitter is useful in situations where you want more precision on the
low end than an ExponentialFitter with exponent base 2 provides without the
hassle of dealing with non-integer boundaries, such as would be created by an
exponential fitter with a base of less than 2. Fibonacci fitters are ideal for
integer metrics that are bounded across a certain range, e.g. integers between 1
and 1,000.

This also cleans up some unit test comments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156773367
2017-05-23 17:22:49 -04:00
nickfelt
9ee9229456 Fix message for SuperordinateDomainInPendingDeleteException
The StatusProhibitsOperationException superclass constructor just wants a raw message to return to the client, so we should pass it an explanation rather than just the superordinate domain name.  In fact, I think the superordinate domain name isn't really necessary since it should be obvious from the hostname in the create/update request what the superordinate domain would be.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156747185
2017-05-23 17:22:49 -04:00
mountford
fb21946366 Fix parameter problem in GenerateEscrowDepositCommand
It was setting the TLD and watermark date parameters as comma-separated lists. What it really needs to do is set the parameters multiple times, once for each value.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156741051
2017-05-23 17:22:49 -04:00
nickfelt
65aaeccfc6 Fix TaskQueueHelper param matching for pull queue tasks
This fixes TaskQueueHelper methods and MatchableTaskInfo so that the .param() matching works for pull queues, by parsing the payload for URL-encoded parameters more liberally.  As such, it updates all the places where formerly we were hacking around this by manually constructing the expected payloads and using TaskMatcher.payload() instead.

It also adds a TaskQueueHelper.assertTasksEnqueued() overload that accepts an Iterable<TaskStateInfo> so that you can cleanly assert that a queue contains the same tasks that were returned via a previous call to getQueueInfo("queue").getTaskInfo().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156604901
2017-05-23 17:22:49 -04:00
jianglai
3f5735ed4b Add Carriage Return in WHOIS disclaimer text
WHOIS disclaimer is read from the config yaml file directly and the line breaks
within the text does not contain carriage return, which ICANN requires. This
CL fixes the non-compliance.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156359526
2017-05-23 17:22:49 -04:00
cgoldfeder
c9d7e75946 Cache Registrars in memory
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.

While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
2017-05-17 12:26:11 -04:00
mountford
74a0defef3 Add the ability to generate RDE deposits in lenient mode
We will not want to run this under normal circumstances, but in cases (such as PDT testing in sandbox) where it's desirable to generate an escrow deposit even if it isn't technically valid XML, this gives us that option. Manual-mode RDE generation is also changed so that, if no watermark date is specified, it defaults to the previous midnight, to better support running of RDE in sandbox to catch data problems.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156108295
2017-05-17 12:22:49 -04:00
jianglai
35624ace08 Require currency specific billingAccountId for registrar
For the currency specific billing account ids to work properly, any REAL registrar that is allowed to register domains in a certain TLD must have a billing account id for the currency of that TLD.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864711
2017-05-17 12:09:46 -04:00
jianglai
6e6a340113 Allow multiple domain creation in nomulus tool
This helps creating test domains. Also fixed a bunch of bad test cases.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864065
2017-05-17 12:08:10 -04:00
mcilwain
ef1487cb57 Make name and address fields required on Registrar
The absence of these fields causes RDE failures, so they are in effect
required on any functioning registry system. We are currently
experiencing problems in sandbox caused by null values on these fields.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155474895
2017-05-17 11:44:46 -04:00
jianglai
8892656722 Make WHOIS domain query return conform to ICANN CL&D policy
ICANN's policy can be found here:
https://www.icann.org/resources/pages/rdds-labeling-policy-2017-02-01-en

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155375811
2017-05-17 11:38:28 -04:00
jianglai
2846f9c6b9 This CL include changes in the registrar console that makes it possible to designate an abuse contact in domain WHOIS record, per ICANN's CL&D requirement.
Frontend validation: ensures that only one WHOIS abuse contact exist per registrar. Any existing WHOIS abuse contact will be overridden when a new one is designated.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155289097
2017-05-17 11:36:53 -04:00
cgoldfeder
275d6ddc10 Disable memcache completely
We've determined that getting correctness semantics right, even
in the few cases that it is possible to do so (see linked bug for
audit) is not worth the bother in terms of highly complicated code
and potential bugs. This CL turns off memcache at the Ofy level
but doesn't rip out the annotations etc. so that we can quickly
turn it back on if this turns out to have been a mistake.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155227761
2017-05-17 11:35:19 -04:00
cgoldfeder
608e121a37 Set the premium cache duration in tests to 60s not 60ms
This was causing test flakiness.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155225717
2017-05-17 11:33:44 -04:00
mmuller
3eef5095d4 Deal with gaps and forks in commit history
If there are gaps or forks in the commit log history files, raise an exception
and display the possible sets of files in the log files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155219410
2017-05-17 11:30:36 -04:00
jianglai
7a3852b5a5 Add visibleInWhoisAsAbuse field in RegistrarContact
Also added corresponding getters and setters for the new field. Note that
nothing has changed on the RDAP front for now, as the CL&D only concerns WHOIS.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155116134
2017-05-17 11:27:24 -04:00
guyben
93c2a1e4f0 Make JSch injection lazy
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154958750
2017-05-17 11:22:53 -04:00
guyben
5e7834b00e Wrap Kms decrypt with Retrier
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154861871
2017-05-03 11:10:53 -04:00
mountford
8bb66dd365 Add delete_host nomulus tool command
This is needed to delete an invalid host resulting from a previous bug.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154757425
2017-05-03 11:08:02 -04:00
mcilwain
e6249196af Fix formatting of DeleteDomainCommand
It was pretty off for some reason, so I just ran
google-java-format on it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154756158
2017-05-03 11:06:19 -04:00
nickfelt
f640d765e8 Log EppExceptions in EppController at INFO (vs FlowRunner at WARNING)
The logging for exceptions in FlowRunner - always at WARNING - has long been sub-optimal.  For EppExceptions it's too aggressive/spammy to log at WARNING because it's generally not actionable - EppException gets properly thrown for all kinds of ordinary reasons (trying to create a resource when one already exists with that foreign key) and/or for client misbehavior that we can't control (sending bad parameter values, etc.).  For non-EppException RuntimeExceptions, it's redundant with existing logging in EppController.

This CL resolves this by removing that logging in FlowRunner entirely in favor of the EppController logging, where we're now logging EppExceptions at INFO in parallel with the existing logging of RuntimeExceptions at SEVERE.  This has the benefit that we're now logging EppExceptions that come from FlowPicker (by way of EppExceptionInProviderException),  which previously were unlogged.

Note however that this does mean that in places where we run FlowRunner without EppController - exclusively test code as it stands today - we'd no longer be logging EppExceptions.  If that seems like a loss, we could either reinstate logging there (at INFO) and just deal with redundant messages for most EppExceptions, or we could add it manually to places where we call FlowRunner.run() in tests and avoid the redundancy that way.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154733365
2017-05-03 11:04:51 -04:00
mountford
11e7374c0f Fix bug which allowed creation of hosts with superordinate domains in pending delete state.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154716883
2017-05-03 11:03:23 -04:00
cgoldfeder
386d2bc6be Make Registrar.loadByClientId explicitly use memcache
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154712979
2017-05-03 11:01:54 -04:00
nickfelt
f9df4cf3f1 Revert unintentional no-op log handler attachment
This snuck in from [] but was just a vestige of when I was testing with the tests still in FlowRunner test and wanted to make sure they still passed with the logging moved to FlowReporter.  Now that the tests are in FlowReporterTest and FlowReporter is mocked out here, attaching the handler to that logger is a no-op.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154601472
2017-05-03 11:00:23 -04:00
mcilwain
876552f1ce Add more EPP metric lifecycle tests
This also refactors some of the existing EPP XML testdata files
to use a consistent wildcard naming scheme.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154569632
2017-05-03 10:58:51 -04:00
cgoldfeder
d4d02f8977 Add a test for Registrar.loadByClientId not being enrolled in a transaction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154470388
2017-05-03 10:52:46 -04:00
mcilwain
8025c7547b Add flow-level EPP metric tests
This also adds an assertion helper for EPP metrics.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154458134
2017-05-03 10:51:12 -04:00
cgoldfeder
f1129ea2b1 Make DomainInfoFlow (and application info) explicitly hit memcache
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
    fail. We aren't ready to do this in prod yet, which is why the tests have a TODO
    in them. The new tests fail if you change line 134 in Ofy to not use memcache
    and either use the unchanged original flow code, or use the new
    inlined code and change loadWithMemcache() to load(). They pass with the new
    inlined code that calls loadWithMemcache(), as long as the @Cache is added to
    DomainResource.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154457655
2017-05-03 10:49:38 -04:00
cgoldfeder
927eb43cbc Un-Ignore a test in DomainApplicationCreateFlowTest
A little injection-foo makes this test possible to run.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154442134
2017-05-03 10:44:56 -04:00
nickfelt
40fa9ff022 Fix some direct uses of ForeignKey(C/H/D)Index
Mostly these are calls to ForeignKeyIndex.create() (a static method) via subclasses, which is pretty misleading in this case since the type of the return value has nothing to do with the subclass you're qualifying the static method call with (the returned type depends only on the type of the EppResource parameter).

Note however though that while the style guide indeed prohibits qualifying static member references with things other than the class name, the subclassing case is apparently not considered subject to that prohibition in general:
https://groups.google.com[]d/msg/java-style/8ViX-Rh2_sc/48n2lz5nAAAJ

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154331605
2017-05-03 10:40:21 -04:00
mcilwain
d30f9411d8 Require that DNS writer be set on Registry entities
We ran into a bunch of prober deployment issues this past week when
attempting to spin up a new cluster because the newly created prober
TLDs had null values for the dnsWriter field. Given that VoidDnsWriter
exists, we can require that dnsWriter always be set, and have people
use that if DNS publishing is not required.

Also cleans up a bunch of related inconsistent exception messages and
tests not verifying said exception messages properly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154325830
2017-05-03 10:38:49 -04:00
cgoldfeder
d7da112c19 Put back accidentally deleted assertion
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154300634
2017-05-03 10:35:44 -04:00
cgoldfeder
3f9501e469 Remove obsolete comment.
init() is from pre-flattening days

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154256803
2017-04-26 11:21:35 -04:00
mountford
e00b0440b2 Roll back accidental inclusion of non-public Mockito helper methods
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154232020
2017-04-26 11:19:59 -04:00
cgoldfeder
4eba2ea07a Make Registry.get(tld) hit memcache explicitly.
TESTED=The test fails if you change line 134 in Ofy to not use memcache
    and use the unchanged original Registry.get() code. This is the
    expected behavior.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154226534
2017-04-26 11:18:02 -04:00
cgoldfeder
9e61f1d6ef Make failfastForCreate for domain and application creates explicitly hit memcache
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
    fail. We aren't ready to do this in prod yet, which is why the tests are still
    marked @Ignore. The new tests fail if you change line 134 in Ofy to not use memcache
    and either use the unchanged original DomainCreateFlow code, or use the new
    inlined code and change loadWithMemcache() to load(). They pass with the new
    inlined code that calls loadWithMemcache(), as long as the @Cache is added to
    DomainResource.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154224748
2017-04-26 11:16:33 -04:00
mountford
cb145e0721 Remove some unnecessary uses of MockitoJUnitRunner, which is discouraged
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154202151
2017-04-26 11:15:07 -04:00
cgoldfeder
51cfc12c1d Add MemcacheHelper to simplify memcache usage in tests
And make two callsites use it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154189438
2017-04-26 11:12:17 -04:00
jianglai
33fabfff04 Remove abuse contact requirement
The requirement will instead be enforced on registrar contact that is visible in WHOIS as abuse contact.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154102111
2017-04-26 11:07:48 -04:00
cgoldfeder
b6e50f0467 Turn on memcache in flow tests.
No tests break because of this, but some tests that were previously
intentionally clearing memcache (for example, the test in
DomainInfoFlow that counts datastore calls) are now doing something
that is not a no-op anymore.

Also fix an incorrect comment about memcache in DatastoreHelper.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154087182
2017-04-26 11:04:56 -04:00