Commit graph

982 commits

Author SHA1 Message Date
guyben
75bb1aacdc Update cloudkms_java with the non-beta version
Changed [] to use v1 instead of v1beta1, and replaced v1beta1 with v1 in all the java files.

If there is special build rules for open-source etc. that also need to be updated, or non "TAP-able" tests that need to be run, please check and see if they are OK.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157895888
2017-06-05 18:17:09 -04:00
mountford
3bbf356497 Fix bug in registrar contact nomulus command
The command was set up such that an update without any contact types specified would clear out the list, instead of leaving them unchanged, as it should.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157766429
2017-06-05 18:17:09 -04:00
mountford
6cc3ee5e15 Add option to registrar_contact command to modify the visible_in_domain_whois_as_abuse flag
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157588734
2017-06-05 18:17:09 -04:00
mountford
009522d078 Change auth settings on RdeStagingAction
[] opened up the settings on RdeStaging, in order to make it usable by the nomulus tool. But in retrospect, we think that all we needed to do was support the POST method, not loosen the auth settings, since nomulus invokes RdeStaging via a task queue. Removing the looser auth settings will bring this action into line with other backend actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157262595
2017-06-05 18:17:09 -04:00
mountford
bc63d80a5e Change auth settings on some tools service actions
This brings the affected actions into line with the settings on other similar actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157259842
2017-06-05 18:17:09 -04:00
mountford
e46937beac Add JS deprecation error suppression
Our build got broken by the deprecation of goog.structs.Map in [] This is a quick fix.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157236744
2017-06-05 18:17:09 -04:00
mmuller
b733443eab Create nomulus tool restore command
Create a nomulus tool restoreCommitLogs command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157103177
2017-06-05 18:17:09 -04:00
mountford
ddb0f5692e Add golden files for request component routing maps
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157026188
2017-06-05 18:17:09 -04:00
mcilwain
bb67841884 Add metrics for async batch operation processing
We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:

* An incrementable metric for each async task processed (split out by
  type of task and result).
* Two event metrics for processing time between when a task is enqueued
  and when it is processed -- tracked separately for contact/host
  deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
  run (this is usually 0). Tracked separately for contact/host deletion
  and DNS refresh on host rename.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
2017-06-05 18:17:09 -04:00
mcilwain
1adeb57fea Add a Fibonacci fitter for metrics bucketing
A Fibonacci fitter is useful in situations where you want more precision on the
low end than an ExponentialFitter with exponent base 2 provides without the
hassle of dealing with non-integer boundaries, such as would be created by an
exponential fitter with a base of less than 2. Fibonacci fitters are ideal for
integer metrics that are bounded across a certain range, e.g. integers between 1
and 1,000.

This also cleans up some unit test comments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156773367
2017-05-23 17:22:49 -04:00
mmuller
e6af34301d Move restore from backend to tools
Move the "restoreCommitLogs" command from the backend module to the tools
module so it's easier to access with nomulus.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156768389
2017-05-23 17:22:49 -04:00
nickfelt
9ee9229456 Fix message for SuperordinateDomainInPendingDeleteException
The StatusProhibitsOperationException superclass constructor just wants a raw message to return to the client, so we should pass it an explanation rather than just the superordinate domain name.  In fact, I think the superordinate domain name isn't really necessary since it should be obvious from the hostname in the create/update request what the superordinate domain would be.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156747185
2017-05-23 17:22:49 -04:00
mountford
fb21946366 Fix parameter problem in GenerateEscrowDepositCommand
It was setting the TLD and watermark date parameters as comma-separated lists. What it really needs to do is set the parameters multiple times, once for each value.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156741051
2017-05-23 17:22:49 -04:00
mcilwain
42513f41ca Remove migration @OnLoad for EppResource StatusValue LINKED
This was needed to correct bad data (LINKED status values on EppResources). The code has been fixed to no longer persist LINKED on any resources and I ran a resave all action yesterday to remove all persisted LINKED status values, so the migration @OnLoad can be safely removed now.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156580334
2017-05-23 17:22:49 -04:00
shikhman
2b1ab579dc Move metrics shutdown behavior to a ShutdownHook
HttpServlet#destroy() is not called on instance shutdown, so the metrics code was never shut down correctly before.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156360866
2017-05-23 17:22:49 -04:00
jianglai
3f5735ed4b Add Carriage Return in WHOIS disclaimer text
WHOIS disclaimer is read from the config yaml file directly and the line breaks
within the text does not contain carriage return, which ICANN requires. This
CL fixes the non-compliance.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156359526
2017-05-23 17:22:49 -04:00
shikhman
413a634533 Automated g4 rollback of changelist 156106183.
*** Reason for rollback ***

This change is crashing alpha

*** Original change description ***

Change metrics ThreadFactory to produce daemon threads

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156336467
2017-05-17 15:19:10 -04:00
mountford
7d5a67f5a2 Localize a variable
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156187775
2017-05-17 12:27:43 -04:00
cgoldfeder
c9d7e75946 Cache Registrars in memory
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.

While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
2017-05-17 12:26:11 -04:00
blickly
9a48aae107 Add a few missing type annotations to braintree externs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156148567
2017-05-17 12:24:26 -04:00
mountford
74a0defef3 Add the ability to generate RDE deposits in lenient mode
We will not want to run this under normal circumstances, but in cases (such as PDT testing in sandbox) where it's desirable to generate an escrow deposit even if it isn't technically valid XML, this gives us that option. Manual-mode RDE generation is also changed so that, if no watermark date is specified, it defaults to the previous midnight, to better support running of RDE in sandbox to catch data problems.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156108295
2017-05-17 12:22:49 -04:00
shikhman
e1f4df86bd Change metrics ThreadFactory to produce daemon threads
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156106183
2017-05-17 12:21:12 -04:00
mmuller
5ff56ac7a0 Make "throws" clause use IOException
Now that we've updated the oauth java6 package in the open source build, we
can safely replace "Exception" with "IOException" in the throws clause of the
method that calls AuthorizationCodeInstalledApp.authorize().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156057190
2017-05-17 12:17:40 -04:00
mmuller
fc5546ad58 Add latest oauth/{java6,jetty} and fix repo procedure
Upgrade to the latest versions of omit_com_google_oauth_client_java6 and
omit_com_google_oauth_client_jetty and fix the broken stuff in our local
mirroring procedures.  Specifically:

- Document that users need to add a GCS url.
- Document how to run the check_bazel_deps script.
- Document that we don't use the in-piper repo.
- Remove the broken ibiblio link for the "truth" package.
- Delete sync_bazel_deps (which doesn't currently work, isn't python-style
  compliant and is no longer relevant since we don't use the in-piper repo)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155898485
2017-05-17 12:12:56 -04:00
jianglai
35624ace08 Require currency specific billingAccountId for registrar
For the currency specific billing account ids to work properly, any REAL registrar that is allowed to register domains in a certain TLD must have a billing account id for the currency of that TLD.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864711
2017-05-17 12:09:46 -04:00
jianglai
6e6a340113 Allow multiple domain creation in nomulus tool
This helps creating test domains. Also fixed a bunch of bad test cases.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864065
2017-05-17 12:08:10 -04:00
larryruili
b08e317505 Add scrap tool for populating required registrar fields
This tool finds all registrars that have a null name, or a null localizedAddress field, and populates them as follows:

- The registrar's name is updated to its clientId
- The registrar's empty localizedAddress fields are replaced with fake data

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155739946
2017-05-17 12:01:49 -04:00
nickfelt
6b6b8d25c7 Add type to RegistrarData billing BigQuery view
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155682529
2017-05-17 12:00:09 -04:00
guyben
c2094eda40 Make EncryptEscrowDepositCommand a RemoteApiCommand
Moving to KmsKeyring, any command that uses the Keyring must be a
RemoteApiCommand to work.

TESTED: successfully encrypted a file with no logged errors.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155673003
2017-05-17 11:58:35 -04:00
mcilwain
6e3a99a16b Tune max # of frontend instances down to 4 in sandbox and alpha
My continuing investigations into necessary resources for running
our environments seems to indicate that four instances should be
sufficient for our purposes.  If it's not, we can always revert.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155607688
2017-05-17 11:55:30 -04:00
nickfelt
2b78571c72 Replace literal 'latest_snapshot' in invoice SQL query
This should be %SOURCE_DATASET% so that it's adjusted appropriately based on the parameter value.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155582718
2017-05-17 11:53:56 -04:00
mmuller
70e3271319 Add "toTime" parameter to RestoreCommitLogsAction web form
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155500640
2017-05-17 11:46:18 -04:00
mcilwain
ef1487cb57 Make name and address fields required on Registrar
The absence of these fields causes RDE failures, so they are in effect
required on any functioning registry system. We are currently
experiencing problems in sandbox caused by null values on these fields.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155474895
2017-05-17 11:44:46 -04:00
jianglai
5313ca58d6 Setup auth settings in PublishDetailReports @Action to support tool access
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155459413
2017-05-17 11:43:09 -04:00
jart
d419f6ac11 Export annotations from Guava
These annotations are part of guava's API, and they cannot safely be
optimized off the compile-time classpath.

This works around google/error-prone#615.

See []
H/T @cushon

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155433554
2017-05-17 11:40:02 -04:00
jianglai
8892656722 Make WHOIS domain query return conform to ICANN CL&D policy
ICANN's policy can be found here:
https://www.icann.org/resources/pages/rdds-labeling-policy-2017-02-01-en

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155375811
2017-05-17 11:38:28 -04:00
jianglai
2846f9c6b9 This CL include changes in the registrar console that makes it possible to designate an abuse contact in domain WHOIS record, per ICANN's CL&D requirement.
Frontend validation: ensures that only one WHOIS abuse contact exist per registrar. Any existing WHOIS abuse contact will be overridden when a new one is designated.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155289097
2017-05-17 11:36:53 -04:00
cgoldfeder
275d6ddc10 Disable memcache completely
We've determined that getting correctness semantics right, even
in the few cases that it is possible to do so (see linked bug for
audit) is not worth the bother in terms of highly complicated code
and potential bugs. This CL turns off memcache at the Ofy level
but doesn't rip out the annotations etc. so that we can quickly
turn it back on if this turns out to have been a mistake.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155227761
2017-05-17 11:35:19 -04:00
cgoldfeder
608e121a37 Set the premium cache duration in tests to 60s not 60ms
This was causing test flakiness.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155225717
2017-05-17 11:33:44 -04:00
mmuller
3eef5095d4 Deal with gaps and forks in commit history
If there are gaps or forks in the commit log history files, raise an exception
and display the possible sets of files in the log files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155219410
2017-05-17 11:30:36 -04:00
jianglai
7a3852b5a5 Add visibleInWhoisAsAbuse field in RegistrarContact
Also added corresponding getters and setters for the new field. Note that
nothing has changed on the RDAP front for now, as the CL&D only concerns WHOIS.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155116134
2017-05-17 11:27:24 -04:00
mountford
354e1fb8b2 Change messages about failed authentication to be warnings instead of info messages
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155114145
2017-05-17 11:25:54 -04:00
guyben
93c2a1e4f0 Make JSch injection lazy
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154958750
2017-05-17 11:22:53 -04:00
guyben
5e7834b00e Wrap Kms decrypt with Retrier
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154861871
2017-05-03 11:10:53 -04:00
nickfelt
7f2821cd7e Open up RdeStagingAction @Action settings to support tool access
This change means that RdeStagingAction now accepts POST requests and supports the API auth method, which is needed for the new GenerateEscrowDepositCommand (as of [] to be able to invoke the action in manual mode.  Note that this shouldn't affect the cron setting that invokes it currently, which are GET requests with INTERNAL auth.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154764870
2017-05-03 11:09:29 -04:00
mountford
8bb66dd365 Add delete_host nomulus tool command
This is needed to delete an invalid host resulting from a previous bug.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154757425
2017-05-03 11:08:02 -04:00
nickfelt
f640d765e8 Log EppExceptions in EppController at INFO (vs FlowRunner at WARNING)
The logging for exceptions in FlowRunner - always at WARNING - has long been sub-optimal.  For EppExceptions it's too aggressive/spammy to log at WARNING because it's generally not actionable - EppException gets properly thrown for all kinds of ordinary reasons (trying to create a resource when one already exists with that foreign key) and/or for client misbehavior that we can't control (sending bad parameter values, etc.).  For non-EppException RuntimeExceptions, it's redundant with existing logging in EppController.

This CL resolves this by removing that logging in FlowRunner entirely in favor of the EppController logging, where we're now logging EppExceptions at INFO in parallel with the existing logging of RuntimeExceptions at SEVERE.  This has the benefit that we're now logging EppExceptions that come from FlowPicker (by way of EppExceptionInProviderException),  which previously were unlogged.

Note however that this does mean that in places where we run FlowRunner without EppController - exclusively test code as it stands today - we'd no longer be logging EppExceptions.  If that seems like a loss, we could either reinstate logging there (at INFO) and just deal with redundant messages for most EppExceptions, or we could add it manually to places where we call FlowRunner.run() in tests and avoid the redundancy that way.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154733365
2017-05-03 11:04:51 -04:00
mountford
11e7374c0f Fix bug which allowed creation of hosts with superordinate domains in pending delete state.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154716883
2017-05-03 11:03:23 -04:00
cgoldfeder
386d2bc6be Make Registrar.loadByClientId explicitly use memcache
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154712979
2017-05-03 11:01:54 -04:00
mountford
44546a3480 Throw error in get_history_entries if the specified ID is invalid
Currently, if the ID is invalid, parentKey is set to null, causing it to return all history entries. Note that there is still a problem that you cannot look up history entries for entities which have been soft deleted, because the foreign key lookup won't work. That is unfortunate, but at least this simple fix makes things better than they were.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154564262
2017-05-03 10:57:20 -04:00