Commit graph

2714 commits

Author SHA1 Message Date
jianglai
744c814bdb Rename METADATA file to README.md
Get rid of unnecessary information that is only useful for Google internal build.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177846457
2017-12-04 15:07:37 -05:00
jianglai
1c1f95992a Move backported JUnit file to third_party (part 2)
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
2017-12-02 11:37:46 -05:00
jianglai
8ef4fbe9d5 Move backported JUnit file to third_party
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
2017-12-01 22:14:06 -05:00
jianglai
82382aaca6 Move jsch and objectify to directly under third_party
We used to have a lot more vendored third_party dependencies, but now we're only going to have three that we know of, so this java/javatests distinction isn't necessary.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177631319
2017-12-01 22:14:06 -05:00
guyben
b8970bfe1b Rename all testdata loading files to conform to a single naming scheme
The scheme is:
- loadBytes: returns a ByteSource of the data
- loadFile: returns a string using UTF8 encoding, optionally applying
  substitutions

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177606406
2017-12-01 22:14:06 -05:00
mcilwain
ebfa27b3ad Require fee extension when registering domain in EAP
Failing to use the fee extension during EAP can result in charges to registrars
that are radically different than what they may have been expecting.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177597883
2017-12-01 22:14:06 -05:00
mcilwain
087a500093 Use Java 8 streams in RdeCounter
This also improves performance by calculating a set once rather than
on-demand every time it is used.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177471572
2017-12-01 22:14:06 -05:00
guyben
68768a561f Refactor EppToolVerifier to accept chaining verify commands
We're doing this to allow several new tests:
- xml files (that exist today)
- xml files with substitutions
- xml content (maybe? Currently private. Caching the files seems more readable)
- no data at all

Instead of having only one interface

eppToolVerifier.verifySent("file1.xml", "file2.xml");

we're refactoring to allow:
eppToolVerifier
  .verifySent("file1.xml")
  .verifySentAny() // we don't care about this epps
  .verifySent("file2.xml", substitutions)
  .verifyNoMoreSent();

In this case we're checking that "exactly 3 EPPs were sent, where the 1st one has content from file1.xml, and the 3rd one has the content from file2.xml, after the given substitutions were applied"

This also updates EppToolCommandTestCase to have only one EppToolVerifier, and
always finish by checking verifyNoMoreSent, meaning that in every test - all
sent epps must be accounted for (verified or skiped)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177353887
2017-12-01 22:14:06 -05:00
guyben
0e3d050dae Temporarily disable deleteProberData cron job in sandbox for load-testing
Loadtesting data is identified as "prober data" by this job (it removes
anything under ".test", not only prober data)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177309096
2017-12-01 22:14:06 -05:00
jianglai
e73325edcf Make FOSS build work with DummyKeyringModule
DummyKeyringModule provides a fake string as the JSON credential used to instantiate a GoogleCredential. Of course this would not work and when the metric reporter requests a GoogleCredential in the main thread. This causes the FOSS build to crash on startup, because it defaults to use DummyKeyringModule.

This change allows a graceful handling of such an error by wrapping any calls to instantiate a metric reporter in a try block. Note that any attempt to write to stackdriver will still fail, but that happens in a different thread and will not make the whole program crash.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177183337
2017-12-01 22:14:06 -05:00
mcilwain
e2db3f914e Clean up some code quality issues
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
2017-12-01 22:14:06 -05:00
mcilwain
0935ba6450 Revert poll message ID changes (going with a different approach)
*** Reason for rollback ***

Going with a safer approach to using fresh poll message IDs that doesn't mutate domains themselves.

*** Original change description ***

Use PollMessage IDs that are globally unique across all time

The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177081870
2017-12-01 22:14:06 -05:00
guyben
38b2cb13bf Add metric for lock life duration
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177070996
2017-12-01 22:14:06 -05:00
jianglai
25b49c57cd Update to bazel 0.8.0
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177040918
2017-12-01 22:14:06 -05:00
guyben
95647528b8 Clean up test "load resources" methods and classes
There is a big mix of different "load Resources" from different libraries
depending on where you were and what type of resource you want. Now there is a
clear hirarchy:

ResourceUtils:
  for use in actual (non-test) code, reads a file from a context directory

TestDataHelper (uses ResourceUtils):
  for use in tests, reads a file from a context directory + "/testdata". Also
  caches the resource so calling it multiple times with the same file will
  not read the file multiple times.

Library specific helpers (e.g. ToolsTestData) (uses TestDataHelper):
  for use in that library's tests only, reads from a specific testdata directory.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177027533
2017-12-01 22:14:05 -05:00
mcilwain
bbe2584da4 Refactor Guava functional methods to use lambdas
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177027488
2017-12-01 22:14:05 -05:00
jianglai
2ae496bfce Export hamcrest packages from junit and mockito to fix build
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176728505
2017-11-23 01:09:07 -05:00
mountford
d75071f503 Log RDAP output formatting failures
Also, a couple tests are reworked to make their output more understandable in
case of failure.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
2017-11-23 01:09:07 -05:00
jianglai
a92cdbe8c3 Use backported JUnit exceptThrows and assertThrows in metrics library
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176584707
2017-11-23 01:09:07 -05:00
mcilwain
6eb0d8689d Add domain_unlock nomulus command
This command is used by registry operators to apply registry locks to
domain names.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176577240
2017-11-23 01:09:07 -05:00
jianglai
7e42ee48a4 Open source GCP proxy
Dagger updated to 2.13, along with all its dependencies.

Also allows us to have multiple config files for different environment (prod, sandbox, alpha, local, etc) and specify which one to use on the command line with a --env flag. Therefore the same binary can be used in all environments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176551289
2017-11-21 19:19:03 -05:00
mcilwain
c7484b25e0 Automatically refactor some exception testing to use new JUnit rules
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176550995
2017-11-21 18:56:04 -05:00
mcilwain
f041b1bac0 Add domain_lock nomulus command
This command is used by registry operators to apply registry locks to
domain names.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176549874
2017-11-21 18:54:42 -05:00
mountford
8cd3979385 Add formatOutput parameter to RDAP documentation
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176532296
2017-11-21 18:53:20 -05:00
mountford
67a41273bb Add RDAP pretty-printing option
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176529676
2017-11-21 18:51:56 -05:00
guyben
6f659659ff Simplify the CloudDnsWriter callWithRetry functional
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176512218
2017-11-21 18:49:14 -05:00
jianglai
3c43ece5be Remove duplicate sources in BUILD files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176374018
2017-11-21 18:47:52 -05:00
mountford
582b7bb76a Update RDAP terms of service in config
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176360318
2017-11-21 18:46:31 -05:00
mcilwain
cd314bdc75 Replace many Work and VoidWork usages with lambdas
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176153460
2017-11-21 18:45:12 -05:00
mcilwain
603e0470cc Use PollMessage IDs that are globally unique across all time
The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176149220
2017-11-21 18:43:50 -05:00
mcilwain
cf15b5937c Upgrade to JUnit 4.12
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176147815
2017-11-21 18:42:28 -05:00
jianglai
0796a0ff1c Check the host is under registry suffix instead of public suffix
Guava now has support to distinguish a registry suffix from a public suffix. Since we are only interested in registrable domains, registry suffix is the proper thing to check.

See:

692446a303/guava/src/com/google/common/net/InternetDomainName.java

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176126916
2017-11-21 18:41:06 -05:00
jianglai
9ab68613a0 Replace FailureStrategy with FailureMetadata in gtld's Subjects
Also changed truth.SubjectFactory to truth.Subject.Factory (plain renaming) and use method reference instead of anonymous class to create the factory when applicable.

FailureMetadata, an opaque object to its users, is introduced to replace FailureStrategy in in custom Subject in order to resolve some existing flaws of FailureStrategy as well as enable new features to be added to Truth.

New API is available in Truth-0.36, if there is a build/pom.xml, it's also updated to use this version.

More information: See []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176110788
2017-11-21 18:39:45 -05:00
mountford
6083ef9ba0 Add RDAP entity search metric information
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175984429
2017-11-21 18:38:22 -05:00
mountford
29913cf5bd Suppress a logging statement most of the time
Most of the time, we don't expect incoming requests to have an authorization
header. So this statement gets printed a lot, and doesn't provide much useful
information. We already have a statement listing what type of
authentication/authorization is required by the endpoint, and other statements
indicating either that authorization was successful with a particular method or
was not successful at all.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175969652
2017-11-21 18:37:00 -05:00
mountford
d840180f3a Add RDAP nameserver search metric information
Also, login logic pulled out to helper methods in the test class.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
2017-11-21 18:35:37 -05:00
jianglai
e4c5024400 Install DummyKeyringModule in FOSS build
Per discussions here:

https://groups.google.com/forum/#!topic/nomulus-discuss/ylDW2PblL60

Any use of keyring in the FOSS build would result in crashes because KMS is not configured. We should use the dummy keyring instead so that a vanilla FOSS deployment to GAE can run. Of course users would still need to configure their keyrings (and revert back to KMS keyring module) when they actually use any of the keys.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175868399
2017-11-21 18:34:13 -05:00
mountford
8521c69caa Add metrics for RDAP domain search
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
2017-11-21 18:31:31 -05:00
larryruili
99996121b5 Add end-to-end code to run Beam template from App Engine
This serves as proof-of-concept to verify we can use Beam for our invoice generation use case. Namely, it checks that we can:
- Deploy a Beam template to GCS
- Read from Bigquery within the template
- Run the template from App Engine

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175755390
2017-11-21 18:30:09 -05:00
jianglai
28417b7599 Update generate_workspace link
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175734112
2017-11-21 18:28:43 -05:00
jianglai
054bf4bb3c Add output_licenses to repositories.bzl
https://github.com/bazelbuild/bazel/issues/2286 is resolved.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175729373
2017-11-21 18:27:22 -05:00
mountford
358fe68f09 Add RDAP metrics for non-search endpoints
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.

The default wildcard type is now INVALID rather than NO_WILDCARD.

A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
2017-11-21 18:25:57 -05:00
mmuller
0ffd3553c3 Increase max number of sandbox frontend instances to 8
This mirrors production in hopes of triggering b/67508570 to test the fix.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175295742
2017-11-21 18:24:32 -05:00
larryruili
8f0c37fc4a Delete obsolete ICANN reporting python code
Now that we've successfully ran ICANN reporting in production with the new Java
code, we can safely delete all the code it obsoletes.

I've also added a small README to explain why we no longer have report copies in piper.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175197665
2017-11-21 18:23:10 -05:00
mountford
ff1ab08302 Add RDAP metric class
The code to use this class will come in future CLs, to avoid a huge CL.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
2017-11-21 18:20:26 -05:00
mcilwain
2aa897e698 Remove unnecessary generic type arguments
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175155365
2017-11-21 18:17:31 -05:00
larryruili
8dcc2d6833 Chain ICANN report upload after staging
This converts the upload task from a cron job to a task chained after staging.
This ensures the upload job only occurs when its dependencies are met, and
provides a faster turnaround time to verify both the staging and upload jobs
are complete.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175045489
2017-11-21 18:16:08 -05:00
jianglai
a6f0ab1429 Collect proxy backend metrics
The metrics are collected in a new handler (BackendMetricsHandler), which gets outbound FullHttpRequest and inbound FullHttpResponse. All metrics collected here are backend metrics, i. e. metrics about connection between the proxy and GAE. They also corresponds to the metrics that we are correctly collection from the [] proxy, minus a few that are related to quota management, which are yet to be implemented.

This CL also removed some unnecessary steps to write the frontend protocol name and client hash certificate into backend channel attributes. Instead, BackendMetricsHandler will read these information from the frontend channel attributes directly. This is because backend channel is established after frontend channel, and reading frontend channel attributes from backend handlers always works. On the other hand, there is no easy way to guarantee that the backend channel attributes are writable from frontend handlers, because the channel itself may not have been fully established.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174513334
2017-11-07 17:43:25 -05:00
mountford
5b8ee87ecc Initial, easy changes to support later addition of RDAP metrics
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
2017-11-07 17:39:05 -05:00
mountford
c126c05810 Link RDAP syntax page to Github RDAP user's guide
It has been decided that we will link the RDAP help pages, not to local HTML copies, but to the Github documentation.

A typo and word order error in the Github version of the user's guide are also
corrected.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174350665
2017-11-07 17:37:35 -05:00