Commit graph

1656 commits

Author SHA1 Message Date
guyben
0e3d050dae Temporarily disable deleteProberData cron job in sandbox for load-testing
Loadtesting data is identified as "prober data" by this job (it removes
anything under ".test", not only prober data)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177309096
2017-12-01 22:14:06 -05:00
jianglai
e73325edcf Make FOSS build work with DummyKeyringModule
DummyKeyringModule provides a fake string as the JSON credential used to instantiate a GoogleCredential. Of course this would not work and when the metric reporter requests a GoogleCredential in the main thread. This causes the FOSS build to crash on startup, because it defaults to use DummyKeyringModule.

This change allows a graceful handling of such an error by wrapping any calls to instantiate a metric reporter in a try block. Note that any attempt to write to stackdriver will still fail, but that happens in a different thread and will not make the whole program crash.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177183337
2017-12-01 22:14:06 -05:00
mcilwain
e2db3f914e Clean up some code quality issues
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
2017-12-01 22:14:06 -05:00
mcilwain
0935ba6450 Revert poll message ID changes (going with a different approach)
*** Reason for rollback ***

Going with a safer approach to using fresh poll message IDs that doesn't mutate domains themselves.

*** Original change description ***

Use PollMessage IDs that are globally unique across all time

The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177081870
2017-12-01 22:14:06 -05:00
guyben
38b2cb13bf Add metric for lock life duration
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177070996
2017-12-01 22:14:06 -05:00
jianglai
25b49c57cd Update to bazel 0.8.0
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177040918
2017-12-01 22:14:06 -05:00
guyben
95647528b8 Clean up test "load resources" methods and classes
There is a big mix of different "load Resources" from different libraries
depending on where you were and what type of resource you want. Now there is a
clear hirarchy:

ResourceUtils:
  for use in actual (non-test) code, reads a file from a context directory

TestDataHelper (uses ResourceUtils):
  for use in tests, reads a file from a context directory + "/testdata". Also
  caches the resource so calling it multiple times with the same file will
  not read the file multiple times.

Library specific helpers (e.g. ToolsTestData) (uses TestDataHelper):
  for use in that library's tests only, reads from a specific testdata directory.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177027533
2017-12-01 22:14:05 -05:00
mcilwain
bbe2584da4 Refactor Guava functional methods to use lambdas
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177027488
2017-12-01 22:14:05 -05:00
jianglai
2ae496bfce Export hamcrest packages from junit and mockito to fix build
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176728505
2017-11-23 01:09:07 -05:00
mountford
d75071f503 Log RDAP output formatting failures
Also, a couple tests are reworked to make their output more understandable in
case of failure.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
2017-11-23 01:09:07 -05:00
jianglai
a92cdbe8c3 Use backported JUnit exceptThrows and assertThrows in metrics library
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176584707
2017-11-23 01:09:07 -05:00
mcilwain
6eb0d8689d Add domain_unlock nomulus command
This command is used by registry operators to apply registry locks to
domain names.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176577240
2017-11-23 01:09:07 -05:00
jianglai
7e42ee48a4 Open source GCP proxy
Dagger updated to 2.13, along with all its dependencies.

Also allows us to have multiple config files for different environment (prod, sandbox, alpha, local, etc) and specify which one to use on the command line with a --env flag. Therefore the same binary can be used in all environments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176551289
2017-11-21 19:19:03 -05:00
mcilwain
c7484b25e0 Automatically refactor some exception testing to use new JUnit rules
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176550995
2017-11-21 18:56:04 -05:00
mcilwain
f041b1bac0 Add domain_lock nomulus command
This command is used by registry operators to apply registry locks to
domain names.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176549874
2017-11-21 18:54:42 -05:00
mountford
8cd3979385 Add formatOutput parameter to RDAP documentation
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176532296
2017-11-21 18:53:20 -05:00
mountford
67a41273bb Add RDAP pretty-printing option
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176529676
2017-11-21 18:51:56 -05:00
guyben
6f659659ff Simplify the CloudDnsWriter callWithRetry functional
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176512218
2017-11-21 18:49:14 -05:00
jianglai
3c43ece5be Remove duplicate sources in BUILD files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176374018
2017-11-21 18:47:52 -05:00
mountford
582b7bb76a Update RDAP terms of service in config
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176360318
2017-11-21 18:46:31 -05:00
mcilwain
cd314bdc75 Replace many Work and VoidWork usages with lambdas
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176153460
2017-11-21 18:45:12 -05:00
mcilwain
603e0470cc Use PollMessage IDs that are globally unique across all time
The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176149220
2017-11-21 18:43:50 -05:00
mcilwain
cf15b5937c Upgrade to JUnit 4.12
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176147815
2017-11-21 18:42:28 -05:00
jianglai
0796a0ff1c Check the host is under registry suffix instead of public suffix
Guava now has support to distinguish a registry suffix from a public suffix. Since we are only interested in registrable domains, registry suffix is the proper thing to check.

See:

692446a303/guava/src/com/google/common/net/InternetDomainName.java

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176126916
2017-11-21 18:41:06 -05:00
jianglai
9ab68613a0 Replace FailureStrategy with FailureMetadata in gtld's Subjects
Also changed truth.SubjectFactory to truth.Subject.Factory (plain renaming) and use method reference instead of anonymous class to create the factory when applicable.

FailureMetadata, an opaque object to its users, is introduced to replace FailureStrategy in in custom Subject in order to resolve some existing flaws of FailureStrategy as well as enable new features to be added to Truth.

New API is available in Truth-0.36, if there is a build/pom.xml, it's also updated to use this version.

More information: See []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176110788
2017-11-21 18:39:45 -05:00
mountford
6083ef9ba0 Add RDAP entity search metric information
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175984429
2017-11-21 18:38:22 -05:00
mountford
29913cf5bd Suppress a logging statement most of the time
Most of the time, we don't expect incoming requests to have an authorization
header. So this statement gets printed a lot, and doesn't provide much useful
information. We already have a statement listing what type of
authentication/authorization is required by the endpoint, and other statements
indicating either that authorization was successful with a particular method or
was not successful at all.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175969652
2017-11-21 18:37:00 -05:00
mountford
d840180f3a Add RDAP nameserver search metric information
Also, login logic pulled out to helper methods in the test class.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
2017-11-21 18:35:37 -05:00
jianglai
e4c5024400 Install DummyKeyringModule in FOSS build
Per discussions here:

https://groups.google.com/forum/#!topic/nomulus-discuss/ylDW2PblL60

Any use of keyring in the FOSS build would result in crashes because KMS is not configured. We should use the dummy keyring instead so that a vanilla FOSS deployment to GAE can run. Of course users would still need to configure their keyrings (and revert back to KMS keyring module) when they actually use any of the keys.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175868399
2017-11-21 18:34:13 -05:00
mountford
8521c69caa Add metrics for RDAP domain search
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
2017-11-21 18:31:31 -05:00
larryruili
99996121b5 Add end-to-end code to run Beam template from App Engine
This serves as proof-of-concept to verify we can use Beam for our invoice generation use case. Namely, it checks that we can:
- Deploy a Beam template to GCS
- Read from Bigquery within the template
- Run the template from App Engine

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175755390
2017-11-21 18:30:09 -05:00
jianglai
28417b7599 Update generate_workspace link
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175734112
2017-11-21 18:28:43 -05:00
jianglai
054bf4bb3c Add output_licenses to repositories.bzl
https://github.com/bazelbuild/bazel/issues/2286 is resolved.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175729373
2017-11-21 18:27:22 -05:00
mountford
358fe68f09 Add RDAP metrics for non-search endpoints
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.

The default wildcard type is now INVALID rather than NO_WILDCARD.

A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
2017-11-21 18:25:57 -05:00
mmuller
0ffd3553c3 Increase max number of sandbox frontend instances to 8
This mirrors production in hopes of triggering b/67508570 to test the fix.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175295742
2017-11-21 18:24:32 -05:00
larryruili
8f0c37fc4a Delete obsolete ICANN reporting python code
Now that we've successfully ran ICANN reporting in production with the new Java
code, we can safely delete all the code it obsoletes.

I've also added a small README to explain why we no longer have report copies in piper.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175197665
2017-11-21 18:23:10 -05:00
mountford
ff1ab08302 Add RDAP metric class
The code to use this class will come in future CLs, to avoid a huge CL.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
2017-11-21 18:20:26 -05:00
mcilwain
2aa897e698 Remove unnecessary generic type arguments
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175155365
2017-11-21 18:17:31 -05:00
larryruili
8dcc2d6833 Chain ICANN report upload after staging
This converts the upload task from a cron job to a task chained after staging.
This ensures the upload job only occurs when its dependencies are met, and
provides a faster turnaround time to verify both the staging and upload jobs
are complete.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175045489
2017-11-21 18:16:08 -05:00
jianglai
a6f0ab1429 Collect proxy backend metrics
The metrics are collected in a new handler (BackendMetricsHandler), which gets outbound FullHttpRequest and inbound FullHttpResponse. All metrics collected here are backend metrics, i. e. metrics about connection between the proxy and GAE. They also corresponds to the metrics that we are correctly collection from the [] proxy, minus a few that are related to quota management, which are yet to be implemented.

This CL also removed some unnecessary steps to write the frontend protocol name and client hash certificate into backend channel attributes. Instead, BackendMetricsHandler will read these information from the frontend channel attributes directly. This is because backend channel is established after frontend channel, and reading frontend channel attributes from backend handlers always works. On the other hand, there is no easy way to guarantee that the backend channel attributes are writable from frontend handlers, because the channel itself may not have been fully established.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174513334
2017-11-07 17:43:25 -05:00
mountford
5b8ee87ecc Initial, easy changes to support later addition of RDAP metrics
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
2017-11-07 17:39:05 -05:00
mountford
c126c05810 Link RDAP syntax page to Github RDAP user's guide
It has been decided that we will link the RDAP help pages, not to local HTML copies, but to the Github documentation.

A typo and word order error in the Github version of the user's guide are also
corrected.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174350665
2017-11-07 17:37:35 -05:00
larryruili
eff2266e35 Add apache beam to registry and open source
This is the initial commit of the new billing system, rewritten as an Apache
Beam pipeline. This contains a basic end-to-end pipeline as proof of concept,
and boilerplate for GenerateInvoicesAction, which will eventually be our
automated invoice generation endpoint.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174184171
2017-11-07 17:36:07 -05:00
jianglai
d3254eaaeb Add a convenience method to clear all registered metrics
Also making these methods public so that other test methods can use them.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174074038
2017-11-07 17:34:36 -05:00
jianglai
b0e062d725 Set HTTP header when processing logout request
The proxy can then use this information to terminate client connection. This is conformant to RFC 5734 which requires the server to disconnect upon responding to EPP logout request:

https://tools.ietf.org/html/rfc5734#section-2

We cannot set "Connection: close" because it is stripped away by App Engine:

https://cloud.google.com/appengine/docs/standard/java/how-requests-are-handled#headers_removed

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173904515
2017-11-07 17:30:13 -05:00
larryruili
abf877f5ae Update appengine-testing open source build to 1.9.58
The original impetus for this came from Kokoro build ID 57aa1042-65aa-46eb-9364-23e2bd17b2fd, where cron/TldFanoutActionTest.testSuccess_returnHttpResponse failed due to some strange nondeterminism in the fake task queue. This test isn't flaky in our build, so we assume it's been fixed in later implementations of appengine-testing.

This also adds a domain-registry-maven mirror for truth8 and clarifies the documentation for adding new libraries.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173587282
2017-11-07 17:27:12 -05:00
larryruili
486c348a00 Add reporting cron jobs to production
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173569808
2017-11-07 17:25:46 -05:00
mountford
11a218f9c3 Derive RDAP link paths from the received query
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
2017-11-07 17:24:20 -05:00
mcilwain
30bfcf9c55 Remove more unused Truth8.assertThat() static imports in tests
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173562809
2017-11-07 17:22:56 -05:00
mountford
74873f90c8 Order RDAP domain searches by TLD in domain name order
I am not happy that another index is required, but the Pantheon console shows that domain indexes are much smaller than the other indexes (because there are fewer domains), so it's not adding an appreciable amount of storage space.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173561771
2017-11-07 17:21:26 -05:00