Commit graph

1091 commits

Author SHA1 Message Date
mcilwain
ad73f3d167 Remove more unnecessary "throws" declarations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201243722
2018-06-27 15:28:52 -04:00
mcilwain
8b263baefa Delete MultiplyingCloudDnsWriter
Now that the large zone re-signing test is complete, we no longer need it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
2018-06-27 15:28:06 -04:00
mcilwain
5d80f124ca Remove unnecessary "throws" declarations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201058582
2018-06-18 18:17:56 -04:00
mcilwain
a7256f5edd Update KMS method signatures to standardize on KeyringException
It came up during the review of [] that it doesn't make a lot of sense
for encrypt() and decrypt() to not throw the same kinds of Exceptions,
especially not for the same kind of problem, just because one happens to use a
Retrier in its internal implementation and the other doesn't.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201054057
2018-06-18 18:16:35 -04:00
mcilwain
f971583dc0 Handle timeouts better in lock handler
Rather than just logging a generic TimeoutException, this will say what action
timed out and how long it had been executing for.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201049025
2018-06-18 18:15:11 -04:00
mcilwain
03f8090886 Automated g4 rollback of changelist 199643208.
*** Reason for rollback ***

We suspect that this is breaking RDE more, so we're going to rollout a cherrypick of this reversion.

*** Original change description ***

Upload to GCS before uploading to FTP

Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.

In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201005260
2018-06-18 18:12:18 -04:00
mcilwain
4098487b80 Fix passing of null to some parameters expecting Optional
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200465575
2018-06-18 18:09:21 -04:00
mcilwain
9097a32cc8 Remove web & protocol WHOIS, check API, and RDAP from frontend
These are now handled by the pubapi service and all publicly facing sites that
were using these APIs have already been migrated over.

For documentation on the newly added dispatch.xml file, see:
https://cloud.google.com/appengine/docs/standard/java/config/dispatchref

Note that the --auto_update_dispatch parameter needs to be passed to the
`appcfg update` command in order to apply this new XML file.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200441580
2018-06-18 18:07:53 -04:00
guyben
646c6fa069 Print GAE-UserId even when null
Also explicitly state that contacts missing GAE-UserId can't access the
registrar console

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200402953
2018-06-18 18:06:23 -04:00
guyben
2d4eeae26c Fix admin access to registryAdminClientId when they aren't associated with it
An admin that isn't associated with any clientId, will default to the
registryAdminClientId.

However, if we set the registryAdminClientId as the session's
CLIENT_ID_ATTRIBUTE, the next time we access the server we have a client-id
attribute we aren't associated with - which returns a "403 Registrar Console
access revoked" error (the assumption is - we were associated with it before
but aren't anymore)

To fix this - we just add all admins as "hasAccessTo" registryAdminClientId, even if it's not in the contacts. This will let admins stay on the admin registrar, without affecting where they log-in initially if they are also contacts in different registrars.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200402856
2018-06-18 18:04:56 -04:00
mcilwain
ccdf8c4b26 Use -ROID for contact roids in tests instead of -TLD
-TLD is only used for domain roids.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200109501
2018-06-18 18:00:34 -04:00
larryruili
446617e4e5 Add proper flagging for start-date sunrise billing
This will allow us to check in actual SUNRISE billing policies per launch (15% discount), instead of relying on ad-hoc timestamps.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200077926
2018-06-18 17:59:07 -04:00
Ben McIlwain
8f666ad017 Remove redundant inputXml log in EPP flows
Currently the input XML to an EPP flow is logged twice, once
in FlowRunner and once in FlowReporter.

The log by FlowReporter was used by reporting but this is no
longer the case.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200057153
2018-06-18 17:56:15 -04:00
mcilwain
ddf55005c3 Allow nomulus list_domains to query any number of TLDs
This limit did not exist prior to [] which added the ability to limit
the size of the list. I didn't think that we needed to be able to query more
than 30 TLDs at any one time so I got rid of batching, but it turns out we do
need this ability for domain_watcher. So I'm re-adding batching, which is a
little bit more complicated now that we're also limiting and sorting by creation
time.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199826414
2018-06-18 17:48:44 -04:00
mcilwain
06fc89c6c5 Shard the RDAP tests
These typically take the longest to complete, so sharding them will make them faster.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199796400
2018-06-18 17:47:15 -04:00
guyben
5aeee19699 Send out Lordn during start-date sunrise
Also prevents signed marks from being used in non-sunrise TldStates.

Currently, we send out a Lordn update only when there's a ClaimNotice, or if
we're in end-date sunrise.

But EPPs can contain a SignedMark instead of a ClaimsNotice for trademarked
domains - in which case we aren't sending out Lordn update. This also applies
to start-date sunrises.

We also change the SignedMark behavior for superusers. Currently, if a
mismatched signed mark is given as superuser, we accept it. That causes
problems when we want to send the Lordn update.

Instead - we no longer allow superusers to give a mismatched SignedMark (just
as we don't allow users to give a bad ClaimNotice). A super user can still
create a domain WITHOUT a signed mark - but if one is provided, it must match.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199783411
2018-06-18 17:45:44 -04:00
Ben McIlwain
658f31933c Add metrics for the new Check API
New metrics are necessary because the new API no longer wraps
an EPP flow, therefore does not get metrics for free.

Metrics include
 - An EventMetric for processing time
 - An IncrementableMetric for request count, with
   availability (available/reserved/registered) and
   pricing (standard/premium) fields

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199708592
2018-06-18 17:44:04 -04:00
larryruili
1c71487e5d Set EAP billing event to one-year period
This is consistent with how we treat RESTORE billing events as well- in
general, fees are considered to be amortized over the course of a year (by the
invoicing team).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199684843
2018-06-18 17:42:29 -04:00
guyben
ea08661598 Upload to GCS before uploading to FTP
Currently we encode and upload the deposite to GCS and the FTP server at the
same time. This makes debugging harder as there are many possible points of
failure, some of which are external and some internal.

In this CL we start by encoding + uploading the deposit to GCS, and once
that's done we copy the data from GCS to the FTP server. This will (hopefully)
allow us to distinguish between errors on the FTP server and errors with the
GCS connection.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199643208
2018-06-18 17:39:36 -04:00
jianglai
27fce55654 Restrict contact info commands to owning registrars
Superuser can also execute contact info commands. AuthInfo is no longer checked in the input and always displayed in the output as the only ones who can get a response are the sponsoring registrar and super user.

Also corrected a Javadoc in which '@' should have been escaped (see https://reflectoring.io/howto-format-code-snippets-in-javadoc/)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199521153
2018-06-18 17:36:42 -04:00
larryruili
7487639e62 Enhance transfer logic for transaction reporting
Explicit transfer acks/nacks reverse the roles for transaction reporting
tabulation- this adds a quick check to account for this going forward.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199474444
2018-06-06 15:19:01 -04:00
mcilwain
5c7a20797e Default clientId to registry's registrar in domain lock commands
All domain locks we've processed so far are as a result of the URS process, for
which the clientId is always that of the registry's registrar. So it makes sense
to default to that value, while still retaining the option to specify it if
required in case we ever support registrar-requested registry locks in the
future.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199350120
2018-06-06 15:17:36 -04:00
mcilwain
897690119e Add a nomulus renew_domain command
This may come in handy when dealing with URS requests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199332085
2018-06-06 15:16:09 -04:00
jianglai
643b30d31f Redact WHOIS output instead of removing the fields altogether
See https://www.icann.org/resources/pages/gtld-registration-data-specs-en/#appendixA for details on how certain fields are redacted.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199295355
2018-06-06 15:14:38 -04:00
mcilwain
7c0b8cab0b Add premium support to nomulus create_domain command
Premium prices are automatically detected and set, with an informational
message displayed to the user prior to executing the command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199223541
2018-06-06 15:11:44 -04:00
jianglai
61f6e666b1 Enforce no logging in production environment
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199156367
2018-06-06 15:10:15 -04:00
Ben McIlwain
c61f36502e Add a new check API that does not wrap the domain check EPP flow
Copied class and test from CheckApiAction. All unit tests passing.

Remaining work: add metrics

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198916177
2018-06-06 15:05:30 -04:00
guyben
9d2b1e7572 Consolidate all Set parameter parsing
Currently, we have two different ways to parse a "set" parameter:
key=value1&key=value2&key=value3...
and
keys=value1,value2,value3

This is error prone for several reasons:
- different parts of the code must be "synchronized" to use the same style (the
  place that creates the request, and the place that parses the request)
- for the key=value1&key=value2, we often use the same key name for the single
  value and the set value. This can result in subtle bugs where part of the
  code will successfully read the key assuming there's only one key (and will
  get the first key=value1, ignoring the rest)

Here we transition everything to the keys=value1,value2,value3 method. This one
was chosen because:
- it's shorter
- it's more intuitive for users
- the key name is plural, differentiating it from the singular key=value that
  other requests might need

-----------------------------------

To make sure there are not "transition issues", we will continue to support
(with warnings) the key=value1&key=value2 parameter parsing until we're sure we
haven't forgotten to update any part of the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198810681
2018-06-06 15:04:02 -04:00
jianglai
c0a7bde95e Remove deprecated PublishDetailReportAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198624767
2018-06-06 14:59:30 -04:00
jianglai
70b13596e4 Migrate to Flogger (green)
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.

No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.

Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.

For more information, see []
Base CL: 197826149

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
2018-05-30 12:18:54 -04:00
jianglai
0d2fb3a8f0 Fix legacy logging bug from FlowRunner
We've moved completely to the JSON based reporting framework. The legacy logging statement is only for human consumption, therefore removing the comments. Also fixes a bug where the last argument is not used due to the formatter only expecting 7 arguments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198558998
2018-05-30 12:18:54 -04:00
jianglai
1f1705aaa6 Migrate to flogger (red)
This is a 'red' Flogger migration CL. Red CLs contain changes which are
likely not to work without manual intervention.

Note that it may not even be possible to directly migrate the logger
usage in this CL to the Flogger API and some additional refactoring may
be required. If this is the case, please note that it should be safe to
submit any outstanding 'green' and 'yellow' CLs prior to tackling this.

If you feel that your use case is not covered by the existing Flogger API
please raise a feature request at []and
revert this CL.

For more information, see []
Base CL: 197826149

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198463651
2018-05-30 12:18:54 -04:00
jianglai
4be4fb0082 Migrate to Flogger (yellow)
This is a 'yellow' Flogger migration CL. Yellow CLs should be mostly safe
but include changes that are notable for one reason or another. Manual
intervention may be required to address small issues.

The comments in this CL indicate cases where suggested code changes
should be double checked, or even modified. There may even be cases where
files outside this CL are affected by changes to things such as logger
visibility. However if a change does not have an associated comment then
it should be safe.

For more information, see []
Base CL: 197826149

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097990
2018-05-30 12:18:54 -04:00
mcilwain
92190f8699 Merge the check_domain_fee nomulus command into check_domain
I'm tired of running check_domain only to then realize I needed to run
check_domain_fee instead because the domain ended up being premium. We require
the use of the fee extension to register domains on all of our TLDs anyway so we
might as well always be sending the fee extension when checking domains
too. There's no additional harm in sending along the fee extension (it only
sends back additional information that is often useful), so just make that the
default and remove the extra unnecessary command.

Note that check_domain_claims can't be merged in too because it fundamentally
works differently. It doesn't query the availability of domain names for
registration, just whether they're trademarked.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097326
2018-05-30 12:18:54 -04:00
mcilwain
674a914afc Add flag to delete domain immediately in nomulus too
We've already had the need to do this on a few occasions, so it's best to make
it easy rather than requiring hand-generated XML all the time.

Also normalizes the boolean --registrar_request parameter to not have arity=1.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198093829
2018-05-30 12:18:54 -04:00
mcilwain
8f456bcf64 Clarify when to use cache (or not) when loading premium lists
You don't want to use the cache when loading them for the purposes of updating
them, but you definitely do still want to use the cache when checking the
price of individual domains.

In [] the cache clearing of premium lists on update was removed. This
is a good thing in aggregate because the cache is per-instance and thus
misleading, but it also caused us to not be able to update the same premium
list twice within an hour because the second update would hit a "PremiumList
was concurrently edited" exception, owing to first loading the stale version
from the cache for the purposes of updating it. Now we bypass the cache for
that purpose.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197768142
2018-05-30 12:18:54 -04:00
jianglai
fc60890136 Migrate to internal FormattingLogger in preparation of migration to Flogger
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197744904
2018-05-30 12:18:54 -04:00
jianglai
86dd6bd59e Add ability to show full WHOIS output in nomulus command
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197731992
2018-05-30 12:18:54 -04:00
mcilwain
ac500652ac Add "pubapi" App Engine service for check API, WHOIS, and RDAP
The migration plan is as follows:
1. This CL, which adds the new "pubapi" service that serves the check API, WHOIS, and RDAP.
2a. Update our public facing sites to switch over to use the new service.
2b. (either order) Rewrite the check API to remove dependencies on flows.
3. ... eventually, once the frontend service is no longer being hit by this traffic, remove its handling of these public endpoints.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197716346
2018-05-30 12:18:54 -04:00
jianglai
0cb303ed7f Fix proxy metrics instrumentation bug
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197209531
2018-05-30 12:18:54 -04:00
mcilwain
c989911526 Batch NORDN pull queue task deletions
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.

Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
2018-05-17 21:52:35 -04:00
mcilwain
b8d7d9da29 Don't delete the old premium list yet after updating to a new one
The issue is that the premium list cache is configured to persist for 60
minutes. So after updating the list, checks/creates for up to the next 60
minutes could still be referring to the old list. That's fine and dandy, unless
you delete the old premium list immediately (*bad*), which makes all domains
appear to now be non-premium for as long as the cache lasts. The reason deleting
the premium list entries makes names appear as non-premium is that a load-by-key
existence check with the domain label itself is used to determine if a name is
premium.

I also removed a misleading cache update statement, which doesn't do what it
appears to be doing (it appears to fix this issue) because cache is
instance-level, and so even if the premium list were updated from the frontend
instance only one of 100 instances would have its cache updated. But it's
updated from the tools service anyway, so it's guaranteed to not be a shared
cache with any instance serving EPP traffic.

On a sidenote, I introduced this bug on 2014-10-27 in [] The domain
label list refactor was my Noogler project.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197033604
2018-05-17 21:52:35 -04:00
jianglai
7388958df7 Do not escape WHOIS output
Both WhoisAction and WhoisHttpAction set the HTTP response content type to "text/plain". There is no need to defensively escape the content. In fact, by escaping the content, it creates more problems down the line.

When used in a website, the response should be written into a DOM node by setting the textContent of the node, which automatically escapes the content.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196743398
2018-05-17 21:52:35 -04:00
mcilwain
9c0d3b6db3 Add limit to list_domains command
This allows list_domains to continue working for large TLDs.

TESTED=Deploys to alpha and it works to list the most recently created domains even
on a TLD with a huge number of domains on it (much more than .app has currently).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196717389
2018-05-17 21:52:35 -04:00
guyben
e4f25c08e8 Improve the error when trying to delete the registrant contact
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196708902
2018-05-17 21:52:35 -04:00
mmuller
25d03f239c Allow "encapsulated output" from the shell command
Since the ConsoleReader now controls the display of the prompt, there is no
longer any way for an external program interacting with the nomulus shell to
recognize when the last command issued has been completed.

To remedy this, we introduce an "--encapsulate_output" flag, which causes
standard output and standard error to be wrapped in a class that precedes
all normal lines of output with a prefix ("out: " or "err: ", accordingly)
and allows the command processor to insert a "SUCCESS" or "FAILURE"
line (with no special prefix) after completing the processing of a
command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196702338
2018-05-17 21:52:35 -04:00
larryruili
6cdbde107f Redirect Registrar.referralUrl UI actions to url field
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196597051
2018-05-17 21:52:35 -04:00
mcilwain
de5645abd9 Don't add all claims data to NORDN verify task
This claims data can exceed the maximum size of a task, causing the NORDN upload
to error out.  It also wasn't even being used anyway. This data is already
logged during the upload and there's no reason to log it as well during the
verify, because there is already a unique actionLogId that can be used to tie
the verify task back to the upload task.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196521160
2018-05-17 21:52:35 -04:00
guyben
c25f765fc5 Set the MIME type BEFORE the payload
When we set the payload, it is converted to bytes using the response's
character set. Changing the MIME type later has no effect on the conversion
to bytes, even though it does change the returned MIME type. This results in
bytes that were encoded using one character set while the response reports a
different character set.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196258752
2018-05-17 21:52:35 -04:00
guyben
1787db106d Fix "one version" error
Jetty conflicts with appengine-integration-testing. Replaced one with the other.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196126483
2018-05-17 21:52:35 -04:00