Commit graph

223 commits

Author SHA1 Message Date
mcilwain
d536cef20f Make Registrar load methods return Optionals instead of Nullables
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).

This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
2017-08-01 16:58:59 -04:00
mcilwain
d3e9ebad16 Remove deprecated singular DNS writer field and update tooling
Note that even though the nomulus command line tool now supports multiple
DNS writers for all subcommands, this still won't work quite yet because
the DNS task queue format migration from [] is still in progress.
After next week's push that migration will be complete and we can remove
the final restriction against only having one DNS writer per TLD.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162490399
2017-08-01 16:50:49 -04:00
guyben
8ff1102223 Add the ability to get_keyring_secret the public key from key pairs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162399452
2017-08-01 16:49:29 -04:00
guyben
e224a67eda Change @Auth to an AutoValue, and created a set of predefined Auths
We want to be safer and more explicit about the authentication needed by the many actions that exist.

As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
2017-08-01 16:33:10 -04:00
bbilbo
7d7048ac12 Declare types in Optional.absent() usage
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161865295
2017-08-01 16:26:18 -04:00
bbilbo
9688638c75 Use History Entry type for flows in VerifyOteServlet
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161855429
2017-08-01 16:24:55 -04:00
mcilwain
37f33e5e7a Migrate plural DNS writers field to being the canonical one
After this point all data is migrated to use the new canonical
plural version, and subsequent code changes can be made that use
multiple writers.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161673486
2017-08-01 16:12:42 -04:00
mcilwain
24587491c9 Make re-save environment entities command use batching
This makes it take a lot less time to run (roughly a 10X speedup).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161666391
2017-08-01 16:10:00 -04:00
mountford
3372ed718a Add documentation about OAuth2 client id configuration
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161571961
2017-07-12 11:03:50 -04:00
bbilbo
f721bda16d Update UpdateDomainCommand to use FormattingLogger
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=160299234
2017-07-10 11:26:37 -04:00
mcilwain
dccc99787e Add TLDs parameter to refresh DNS action
This will allow us to migrate one TLD at a time by refreshing all zones
on the specified TLD after dual-writing is enabled.

Note that the TLDs parameter is required, which seems reasonable given
that almost all imagined use cases would be on a by-TLD basis.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=160294546
2017-07-10 11:23:57 -04:00
bbilbo
bbdf9bfc38 Refactor CreateDomainCommand and add UpdateDomainCommand
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=160282921
2017-07-10 11:22:39 -04:00
mcilwain
30d5d05fdf Refactor/rename refresh all DNS action
I'm moving it out of the scrap folder too because there's nothing else
in there and we do want to retain this indefinitely because it's a useful
tool for performing DNS writer migrations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=160168902
2017-07-10 11:18:41 -04:00
nickfelt
c41cf9142a Remove FixDomainTransferDataCommand
Scrap tool is no longer needed.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159500743
2017-06-21 09:53:03 -04:00
nickfelt
fc78fd7d4a Update transfer data fixing command to fix one more domain
I happened to notice that one domain affected by b/33289763 had existing transfer data, so in that case the data got overwritten to look like a server cancellation (a year after the transfer request, so obviously wrong) rather than creating invalid transfer data, and hence wasn't picked up in the previous fixes.  See b/33289763#comment32 for more details.

This CL renames the RemoveDomainTransferDataCommand used to fix the errors to FixDomainTransferDataCommand and changes its behavior to instead reset the TransferData to a server-approval result.  It also changes it to load domains by ROID because loadByForeignKey() fails for deleted domains.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159273225
2017-06-21 09:49:53 -04:00
jianglai
d1ef4b9c37 Allow partial update of billing account map
When doing update_registrar, it is now possible to only specify the currencies and the account ids that need updating.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159262119
2017-06-21 09:48:19 -04:00
mountford
7d2f53a6fe Remove xsrfScope and xsrfProtection authentication attributes
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159121132
2017-06-21 09:46:40 -04:00
cgoldfeder
ae039aa0d8 Remove all vestiges of memcache
Memcache is already off but now it's not in the code anymore.

This includes removing domain creation failfast, since that is actually
slower now than just running the flow - all you gain is a non-transactional
read over a transactional read, but the cost is that you always pay that
read, which is going to drive up latency.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158183506
2017-06-14 10:28:24 -04:00
mountford
3bbf356497 Fix bug in registrar contact nomulus command
The command was set up such that an update without any contact types specified would clear out the list, instead of leaving them unchanged, as it should.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157766429
2017-06-05 18:17:09 -04:00
mountford
6cc3ee5e15 Add option to registrar_contact command to modify the visible_in_domain_whois_as_abuse flag
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157588734
2017-06-05 18:17:09 -04:00
mountford
bc63d80a5e Change auth settings on some tools service actions
This brings the affected actions into line with the settings on other similar actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157259842
2017-06-05 18:17:09 -04:00
mmuller
b733443eab Create nomulus tool restore command
Create a nomulus tool restoreCommitLogs command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157103177
2017-06-05 18:17:09 -04:00
mountford
ddb0f5692e Add golden files for request component routing maps
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157026188
2017-06-05 18:17:09 -04:00
mountford
fb21946366 Fix parameter problem in GenerateEscrowDepositCommand
It was setting the TLD and watermark date parameters as comma-separated lists. What it really needs to do is set the parameters multiple times, once for each value.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156741051
2017-05-23 17:22:49 -04:00
cgoldfeder
c9d7e75946 Cache Registrars in memory
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.

While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
2017-05-17 12:26:11 -04:00
mmuller
5ff56ac7a0 Make "throws" clause use IOException
Now that we've updated the oauth java6 package in the open source build, we
can safely replace "Exception" with "IOException" in the throws clause of the
method that calls AuthorizationCodeInstalledApp.authorize().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156057190
2017-05-17 12:17:40 -04:00
jianglai
35624ace08 Require currency specific billingAccountId for registrar
For the currency specific billing account ids to work properly, any REAL registrar that is allowed to register domains in a certain TLD must have a billing account id for the currency of that TLD.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864711
2017-05-17 12:09:46 -04:00
jianglai
6e6a340113 Allow multiple domain creation in nomulus tool
This helps creating test domains. Also fixed a bunch of bad test cases.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864065
2017-05-17 12:08:10 -04:00
larryruili
b08e317505 Add scrap tool for populating required registrar fields
This tool finds all registrars that have a null name, or a null localizedAddress field, and populates them as follows:

- The registrar's name is updated to its clientId
- The registrar's empty localizedAddress fields are replaced with fake data

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155739946
2017-05-17 12:01:49 -04:00
nickfelt
6b6b8d25c7 Add type to RegistrarData billing BigQuery view
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155682529
2017-05-17 12:00:09 -04:00
guyben
c2094eda40 Make EncryptEscrowDepositCommand a RemoteApiCommand
Moving to KmsKeyring, any command that uses the Keyring must be a
RemoteApiCommand to work.

TESTED: successfully encrypted a file with no logged errors.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155673003
2017-05-17 11:58:35 -04:00
nickfelt
2b78571c72 Replace literal 'latest_snapshot' in invoice SQL query
This should be %SOURCE_DATASET% so that it's adjusted appropriately based on the parameter value.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155582718
2017-05-17 11:53:56 -04:00
mcilwain
ef1487cb57 Make name and address fields required on Registrar
The absence of these fields causes RDE failures, so they are in effect
required on any functioning registry system. We are currently
experiencing problems in sandbox caused by null values on these fields.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155474895
2017-05-17 11:44:46 -04:00
mountford
8bb66dd365 Add delete_host nomulus tool command
This is needed to delete an invalid host resulting from a previous bug.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154757425
2017-05-03 11:08:02 -04:00
mountford
44546a3480 Throw error in get_history_entries if the specified ID is invalid
Currently, if the ID is invalid, parentKey is set to null, causing it to return all history entries. Note that there is still a problem that you cannot look up history entries for entities which have been soft deleted, because the foreign key lookup won't work. That is unfortunate, but at least this simple fix makes things better than they were.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154564262
2017-05-03 10:57:20 -04:00
mcilwain
d30f9411d8 Require that DNS writer be set on Registry entities
We ran into a bunch of prober deployment issues this past week when
attempting to spin up a new cluster because the newly created prober
TLDs had null values for the dnsWriter field. Given that VoidDnsWriter
exists, we can require that dnsWriter always be set, and have people
use that if DNS publishing is not required.

Also cleans up a bunch of related inconsistent exception messages and
tests not verifying said exception messages properly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154325830
2017-05-03 10:38:49 -04:00
guyben
ab515cb352 Replace KeystoreKeyring with KmsKeystore comparison
Replace KeystoreKeyring with ComparatorKeyring between KeystoreKeyring and
KmsKeystore. In the opensource version, will replace DummyKeyring with
KmsKeyring directly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152893767
2017-04-13 10:58:57 -04:00
mountford
dea386d08a Change GenerateEscrowDepositCommand to trigger back end deposit generation
Previously, GenerateEscrowDepositCommand generated the deposit itself. Channeling it through the existing deposit generation code make things more maintainable.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152847950
2017-04-13 10:57:24 -04:00
mountford
4f94464eaf Allow RdeStagingAction to be invoked manually
RdeStagingAction always processed all RDE and BRDA deposits currently outstanding, updating the cursors appropriately and kicking off the upload job. Sometimes we don't want all that. We just want to create a specific deposit by hand, without modifying the cursors or uploading. This CL adds parameters to support that.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152415959
2017-04-10 13:47:51 -04:00
nickfelt
5081d780dc Add assertTldsExist(Iterable<String>) to check multiple TLDs at once
This is better than calling assertTldExists() inside a for loop because you can throw a single exception reporting all bad TLDs at once rather than only getting as far as the first failure.  And then it's also a one-liner instead of 3 lines.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152412876
2017-04-10 13:44:52 -04:00
mountford
5127aeafb5 Enable authentication/authorization checks
The code to authenticate and authorize incoming requests (including via OAuth) has been in the system. This CL actually turns it on, since we are satisfied from logging information that it is not unjustly denying access.

Auth settings are also updated on a few commands missed earlier.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152381820
2017-04-10 13:41:51 -04:00
guyben
47870f98eb Add UpdateKmsKeyringCommand to update keys in the KMS Keyring
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152253868
2017-04-10 13:26:43 -04:00
nickfelt
ef3977e837 Remove unused injected ClientScopeQualifier
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152253768
2017-04-10 13:25:11 -04:00
guyben
c3df4e26a3 Add GetKeyringSecretCommand to retrieve keyring secrets.
Cloned from CL 149476124 by 'g4 patch'.
Original change by shikhman@shikhman:registry-secrets-2:897:citc on 2017/03/07 15:37:09.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151950797
2017-04-05 10:13:45 -04:00
mountford
62c7a3935a Add nomulus tool command to delete a TLD
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151863632
2017-04-05 10:10:38 -04:00
jianglai
ff9c72097c Add Registrar Account Data view in BigQuery snapshot
This adds a new view table that contains the registrar id, the currency-specific billing account id and the corresponding currency in latest_billing dataset based on latest_snapshot dataset.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151363209
2017-04-05 10:06:04 -04:00
jianglai
190be064cb Add billingAccountId to BillingData table
Join RegistrarAccountId table to BillingData to append additional billingAccountId column

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151362466
2017-04-05 10:04:32 -04:00
nickfelt
0d32b6b7b2 Remove MakeBillingTablesCommand credit data views
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151130847
2017-03-27 13:28:33 -04:00
mmuller
5ffb424682 Add a "loadtest" command to nomulus tool
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151034721
2017-03-27 13:16:35 -04:00
nickfelt
aa35d74900 Remove UpdateCreditsCommand
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151023241
2017-03-27 13:12:00 -04:00