These testing helper functions can't be handled by the automatic refactoring
tool because they're taking in expected exception details as parameters.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178832406
This is in preparation for running the automatic refactoring script that
will replace all ExpectedExceptions with use of JUnit 4.13's assertThrows/
expectThrows.
Note that I have recorded the callsites of assertions about EppExceptions
being marshallable and will edit those specific assertions back in after
running the automatic refactoring script (which do not understand these).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178812403
The quotas can be configured in the yaml configuration file. Default quota will be applied to any userId that is not matched in the custom quota list.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178804649
Detailed discussion can be found on github (https://github.com/bazelbuild/bazel/issues/3215). On newer Debian systems as well as on macOS, the temporary folder created to store gpg configs are too long, which results in an error:
gpg: can't connect to the agent: File name too long
This CL makes affected tests use /tmp as temp folder base. Other options are discussed in the github issue, but this one seems to be the mostly non-harmful one. Note that this change only affect the FOSS build, as internal builds are not affected by the temp directory issue, probably due to the internal build tool using a different base temp directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178803914
This fixes a bug where collections of incompatible types are being tested for equality to each other (e.g.: Set<Foo> equals Set<Integer> should never return true unless both sets are empty, a bit of a vacuous assertion).
This change is necessary to unblock future improvements to the static analysis capabilities of the java compiler.
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178798071
It turns out that the RDAP spec does not envision multiple help pages. We can
still support them (for the TOS, for instance), but we shouldn't expect users
to go searching for help other than the main page. Therefore, consolidate the
useful information on the main page, and get rid of some of the others.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178792548
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.
Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
This also cleans up a few miscellaneous code quality issues encountered
while adding the new setter: using a cleaner way to conditionally set field
values, documenting the format of the add grace period parameters, and
improves some code comments and formatting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178387731
We were running FOSS tests on an old version of Ubuntu (14.04) which comes with a rather old gpg version 1.4.16 compared to the current version 2.1.22 (https://lists.gnupg.org/pipermail/gnupg-announce/2017q3/000411.html). As a result some default settings have changed between these versions, leading to test failures when tests are run on newer platforms. In this CL several of the settings are made explicit, no longer depending on default values, which makes them work on either platform.
1. "--no-mdc-error" is set. We do not have mdc integrity protection for the test keys, which results in a non-zero return value for newer versions of GPG. Setting this flag makes return value zero again.
2. "--keyid-format" long is set. GPG key IDs are the last 16 (long key id) or 8 (short key id) octets of the key fingerprint (https://security.stackexchange.com/questions/84280/short-openpgp-key-ids-are-insecure-how-to-configure-gnupg-to-use-long-key-ids-i). Older version uses the short id as default, whereas newer versions defaults to long id. Also change the expected key ID to 16 bytes accordingly.
3. Output stderr in GpgSystemCommandRule when failure occurs during key import. This help debug key import failure due to too long gpg root path. Note: this failure itself is not fixed and still would occur on newer Debian or macOS systems.
4. Set gpg root folder permission correctly to 700, otherwise the newer version gpg will return non-zero value. Previously used method set it to 755.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178163297
Before pushing an update to Cloud DNS, the CloudDnsWriter needs to read all the domain RRSs from Cloud DNS one by one to know what to delete.
Doing so sequentially results in update times that are too long (approx 200ms per domain, which is 20 seconds per batch of 100) severely limiting our QPS.
This CL uses Concurrent threading to do the Cloud DNS queries in parallel. Unfortunately, my preferred method (Set.parallelStream) doesn't work on App Engine :(
This reduces the per-item time from 200ms to 80ms, which can be further reduced to 50ms if we remove the rate limiter (currently set to 20 per second).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178126877
This establishes a fully functional pipeline which generates detail reports for each registrar_tld pair from Bigquery. The main features:
1. Deserialization from AVRO GenericRecord (from Bigquery) into BillingEvent, a POJO we control. This is especially valuable to enable intrinsic type-safety at the start of the pipeline.
2. Addition of .sql files containing the queries used to generate detail reports. These will later be templated to enable general usage.
3. Multi-file-writing within a single TextIO transform, which writes BillingEvents to different files based on their registrar_tld key combo.
This also upgrades the Beam core SDK referenced in repositories.bzl to 2.2.0 and returns the definitions to alphabetical order, to facilitate use of the check_bazel_deps.py script.
The final steps are:
- Converting this to a Nomulus command
- Templating the .sql queries
- @Injecting the @Config values for a given project
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178124838
This solves the problem of external poll message IDs not being globally
unique by simply appending the event year. This means that autorenew poll
messages will increment by one every year, so they will always be unique.
This also requires no data schema changes, and thus most importantly, no
data migration.
Incoming requests lacking this new year field will continue to work for
now for backwards compatibility reasons. This is possible because we don't
actually use the year for anything.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178012685
Currently, if for some reason publishDnsUpdates gets a request to publish
domains to a DnsWriter that doesn't belong to said domain - it logs a warning
but published anyway.
This can happen when Writers are changed (swapped for a different writer)
leaving update commands "stuck" with the wrong writer.
Normally you'd expect these update commands to just publish their data and be
on their way. However, if the update fails for some reason (likely - if the
Writer change happened BECAUSE the updates are failing) then the same
publishDnsUpdate command will continue to run forever.
This CL changes the behavior for "publish to wrong DnsWriter" to instead
requeue the batched domains / hosts back to the Dns-pull queue, allowing them
to be re-batched (and hence published) with the correct DnsWriter(s). This
re-batching will take place in ReadDnsQueueAction.java
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177863076
There has been quite some descriptiveness between github and our internal build. I had to manually push a commit (1c1f95992a) to bring github up-to-date.
Now the github version is identical to what we'd get from doing a -dr-mkfoss. Hopefully the next time things will go smoothly.
The culprit turns out to be MOE itself. It was not attributing changes to commits correctly when the change involves moving files as a result of modifications made to moe_config.json. When moe_config.json is altered in a CL to move files around, MOE always thinks that move happens in the first commit to be pushed.
For example, if we have CL1,CL2,CL3, which correspond to CM1, CM2 and CM3 to be pushed to github, and a change to moe_config.json in CL3 moves a folder, MOE will think that move happens in CM1. This usually is not a problem when all commits are pushed, but when doing manual rebasing and cherry-picking, this will result in unintended dropped changes along with a commit.
I'll do a push to github early next week to confirm that things are back to normal.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177844920
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
The scheme is:
- loadBytes: returns a ByteSource of the data
- loadFile: returns a string using UTF8 encoding, optionally applying
substitutions
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177606406
Failing to use the fee extension during EAP can result in charges to registrars
that are radically different than what they may have been expecting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177597883
We're doing this to allow several new tests:
- xml files (that exist today)
- xml files with substitutions
- xml content (maybe? Currently private. Caching the files seems more readable)
- no data at all
Instead of having only one interface
eppToolVerifier.verifySent("file1.xml", "file2.xml");
we're refactoring to allow:
eppToolVerifier
.verifySent("file1.xml")
.verifySentAny() // we don't care about this epps
.verifySent("file2.xml", substitutions)
.verifyNoMoreSent();
In this case we're checking that "exactly 3 EPPs were sent, where the 1st one has content from file1.xml, and the 3rd one has the content from file2.xml, after the given substitutions were applied"
This also updates EppToolCommandTestCase to have only one EppToolVerifier, and
always finish by checking verifyNoMoreSent, meaning that in every test - all
sent epps must be accounted for (verified or skiped)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177353887
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
*** Reason for rollback ***
Going with a safer approach to using fresh poll message IDs that doesn't mutate domains themselves.
*** Original change description ***
Use PollMessage IDs that are globally unique across all time
The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177081870
There is a big mix of different "load Resources" from different libraries
depending on where you were and what type of resource you want. Now there is a
clear hirarchy:
ResourceUtils:
for use in actual (non-test) code, reads a file from a context directory
TestDataHelper (uses ResourceUtils):
for use in tests, reads a file from a context directory + "/testdata". Also
caches the resource so calling it multiple times with the same file will
not read the file multiple times.
Library specific helpers (e.g. ToolsTestData) (uses TestDataHelper):
for use in that library's tests only, reads from a specific testdata directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177027533
Also, a couple tests are reworked to make their output more understandable in
case of failure.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
This command is used by registry operators to apply registry locks to
domain names.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176577240
Dagger updated to 2.13, along with all its dependencies.
Also allows us to have multiple config files for different environment (prod, sandbox, alpha, local, etc) and specify which one to use on the command line with a --env flag. Therefore the same binary can be used in all environments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176551289
This command is used by registry operators to apply registry locks to
domain names.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176549874
The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176149220
Also changed truth.SubjectFactory to truth.Subject.Factory (plain renaming) and use method reference instead of anonymous class to create the factory when applicable.
FailureMetadata, an opaque object to its users, is introduced to replace FailureStrategy in in custom Subject in order to resolve some existing flaws of FailureStrategy as well as enable new features to be added to Truth.
New API is available in Truth-0.36, if there is a build/pom.xml, it's also updated to use this version.
More information: See []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176110788
Also, login logic pulled out to helper methods in the test class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
This serves as proof-of-concept to verify we can use Beam for our invoice generation use case. Namely, it checks that we can:
- Deploy a Beam template to GCS
- Read from Bigquery within the template
- Run the template from App Engine
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175755390
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.
The default wildcard type is now INVALID rather than NO_WILDCARD.
A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
The code to use this class will come in future CLs, to avoid a huge CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
This converts the upload task from a cron job to a task chained after staging.
This ensures the upload job only occurs when its dependencies are met, and
provides a faster turnaround time to verify both the staging and upload jobs
are complete.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175045489
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
This is the initial commit of the new billing system, rewritten as an Apache
Beam pipeline. This contains a basic end-to-end pipeline as proof of concept,
and boilerplate for GenerateInvoicesAction, which will eventually be our
automated invoice generation endpoint.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174184171
Also making these methods public so that other test methods can use them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174074038
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
