Soy is going to disallow passing params that are unused in the called template and its dependencies. This CL removes these unused params from the call sites.
Passing an unused param might indicate a bug such as having a typo in an optional parameter. Please review this CL carefully and edit the code in Critique if this is the case.
More information: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180571605
This makes a few cosmetic changes that prepares the pipeline for production.
Namely:
- Converts file names to include the input yearMonth, mostly mirroring the original invoicing pipeline.
- Factors out the yearMonth logic from the reporting module to the more common backend module. We will likely use the default yearMonth logic in other backend tasks (such as spec11 reporting).
- Adds the "withTemplateCompatability" flag to the Bigquery read, which allows multiple uses of the same template.
- Adds the 'billing' task queue, which retries up to 5 times every 3 minutes, which is about the rate we desire for checking if the pipeline is complete.
- Adds a shell 'invoicing upload' class, which tests the retry semantics we want for post-generation work (e-mailing the invoice to crr-tech, and publishing detail reports)
While this cl may look big, it's mostly just a refactor and setting up boilerplate needed to frame the upload logic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179849586
ArrayList is more performant and there's no reason to use a LinkedList here.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179717525
Stream.concat only accepts 2 parameters. Streams.concat on the other hand
accepts any number of parameters.
Moving to Streams.concat for all uses (2 or more) makes sense for uniformity
and convenience reasons.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179716648
This creates an end-to-end test that checks for proper billing pipeline IO writes. The only remaining test would be to add a test for the Bigquery query, but see b/70839142 for why I've deemed that more work than worthwhile.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179706561
In Truth8, we can do assertThat(stream) directly. It's less verbose and clearer
in most cases.
Note that for the "finishers" (e.g. "containsExactyElementsIn") - streams are
still not allowed. So when there is:
assertThat(stream.map(someTransformation).collect(toList()))
.containsExactlyElementsIn(expecteStream.map(someTransformation).collect(toList()));
I kept the .collect in the assertThat to preserve the symmetry with the
finisher.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179697587
This also incorporates general improvements and additions to the existing EPP
lifecycle tests around domain deletion. As a refresher: There is a 5 day
add grace period (AGP) following domain creation. Domains that are deleted
during that period have their create costs (but not EAP costs) refunded. This
deletion takes place immediately. Refunds are implemented by issuing a
Cancellation for the associated OneTime billing event.
Domains that are deleted after AGP ends first go through a 30 day redemption
grace period followed by a 5 day pending deletion period. No create fees are
refunded in this case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179597874
This is done first and formost to stop "empty" commits that cause errors in
publishDnsUpdates. The reason being that the Cloud DNS api fails when there are
no updates at all in a change.
Allowing this is a requirement for the writer to be idempotent - if we delete a
domain, then run the writer to delete it again - we'll get 0 additions and 0
deletions which fails.
This isn't theoretical either - we've seen it happen, causing a
publishDnsUpdates to fail over and over again.
While fixing this, we also remove all RRS that are common between additions and
deletions. This is just an optimization and shouldn't affect behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179525218
In addition, while adding the tests, I became discontented with the thoroughness of the cursor navigation tests, which checked only the number of items returned, not their proper ordering. So I updated them to be more careful, and backported the changes to the nameserver and entity search tests as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179442118
That's 50 each for frontend and backend and 5 for tools. Since the
MetricExporter bug has been fixed for awhile now, we aren't gaining anything by
artificially keeping the instance number low, whereas we might benefit from
higher instance counts, e.g. for load-testing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179432038
Right now VoidDnsWriter is injected in the tools - meaning it's possible to
*set* VoidDnsWriter as the writer of a TLD - but it isn't injected in the
backend - meaning we get an error if we actually try to use it.
We need VoidDnsWriter at least for load-testing, and in general for any test
TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179425574
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238852
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238745
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238504
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179221800
These checks were removed in [] and re-adding them is the last
step of the migration to using expectThrows/assertThrows globally.
Note that this is roughly half of them. More to come in a follow-up CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179216707
The assertThrows/expectThrows refactoring script does not use method
references, apparently.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089048
A couple methods were moved to new locations so they are accessible to all types of search queries, not just nameservers like they originally were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089014
This is needed to fix an inability in Java 8 to correctly infer the type
when the transaction was being allowed to return the value it loaded. The
error was:
INFO: Compilation unit has error diagnostics: [third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:109: error: incompatible types: inference variable R has incompatible bounds
ofy().transact(() -> rdeImportUtils.importEppResource(newContact));
^
upper bounds: java.lang.Object
lower bounds: void, third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:132:
error: incompatible types: inference variable R has incompatible bounds
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179082154
we set the "denial of existence" to NSEC (rather than NSEC3), because preventing "walking the zone" isn't an issue for TLDs.
It uses the default security configuration for everything else, which at the time of this writing is:
Key signing: RSASHA256, key length of 2048
Zone signing: RSASHA256, key length of 1024
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179045575
It was easier to simply move these over manually than to try to debug
the automated tooling.
I also changed the case in an exception message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178926365
These testing helper functions can't be handled by the automatic refactoring
tool because they're taking in expected exception details as parameters.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178832406
It's untested, we suspect it's not actually working properly, and we don't
intend to ever need to use anything having to do with auctions ever again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178815580
This is in preparation for running the automatic refactoring script that
will replace all ExpectedExceptions with use of JUnit 4.13's assertThrows/
expectThrows.
Note that I have recorded the callsites of assertions about EppExceptions
being marshallable and will edit those specific assertions back in after
running the automatic refactoring script (which do not understand these).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178812403
The quotas can be configured in the yaml configuration file. Default quota will be applied to any userId that is not matched in the custom quota list.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178804649
Detailed discussion can be found on github (https://github.com/bazelbuild/bazel/issues/3215). On newer Debian systems as well as on macOS, the temporary folder created to store gpg configs are too long, which results in an error:
gpg: can't connect to the agent: File name too long
This CL makes affected tests use /tmp as temp folder base. Other options are discussed in the github issue, but this one seems to be the mostly non-harmful one. Note that this change only affect the FOSS build, as internal builds are not affected by the temp directory issue, probably due to the internal build tool using a different base temp directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178803914
This forks off the results of the billing_events query from Bigquery and generates the overall invoice for the month.
This also turns the ad hoc beam binary into a proper Nomulus command, allowing us to @Inject from the RegsistryToolComponent and thus template the BillingEvent Bigquery query.
Next steps:
- Add the beam package to the open source build
- That will likely require updates to the open source jars, since I'm using features from 2.2.0 which was only just released.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178798943
This fixes a bug where collections of incompatible types are being tested for equality to each other (e.g.: Set<Foo> equals Set<Integer> should never return true unless both sets are empty, a bit of a vacuous assertion).
This change is necessary to unblock future improvements to the static analysis capabilities of the java compiler.
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178798071
It turns out that the RDAP spec does not envision multiple help pages. We can
still support them (for the TOS, for instance), but we shouldn't expect users
to go searching for help other than the main page. Therefore, consolidate the
useful information on the main page, and get rid of some of the others.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178792548
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.
Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
This also cleans up a few miscellaneous code quality issues encountered
while adding the new setter: using a cleaner way to conditionally set field
values, documenting the format of the add grace period parameters, and
improves some code comments and formatting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178387731
We were running FOSS tests on an old version of Ubuntu (14.04) which comes with a rather old gpg version 1.4.16 compared to the current version 2.1.22 (https://lists.gnupg.org/pipermail/gnupg-announce/2017q3/000411.html). As a result some default settings have changed between these versions, leading to test failures when tests are run on newer platforms. In this CL several of the settings are made explicit, no longer depending on default values, which makes them work on either platform.
1. "--no-mdc-error" is set. We do not have mdc integrity protection for the test keys, which results in a non-zero return value for newer versions of GPG. Setting this flag makes return value zero again.
2. "--keyid-format" long is set. GPG key IDs are the last 16 (long key id) or 8 (short key id) octets of the key fingerprint (https://security.stackexchange.com/questions/84280/short-openpgp-key-ids-are-insecure-how-to-configure-gnupg-to-use-long-key-ids-i). Older version uses the short id as default, whereas newer versions defaults to long id. Also change the expected key ID to 16 bytes accordingly.
3. Output stderr in GpgSystemCommandRule when failure occurs during key import. This help debug key import failure due to too long gpg root path. Note: this failure itself is not fixed and still would occur on newer Debian or macOS systems.
4. Set gpg root folder permission correctly to 700, otherwise the newer version gpg will return non-zero value. Previously used method set it to 755.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178163297
Before pushing an update to Cloud DNS, the CloudDnsWriter needs to read all the domain RRSs from Cloud DNS one by one to know what to delete.
Doing so sequentially results in update times that are too long (approx 200ms per domain, which is 20 seconds per batch of 100) severely limiting our QPS.
This CL uses Concurrent threading to do the Cloud DNS queries in parallel. Unfortunately, my preferred method (Set.parallelStream) doesn't work on App Engine :(
This reduces the per-item time from 200ms to 80ms, which can be further reduced to 50ms if we remove the rate limiter (currently set to 20 per second).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178126877
This establishes a fully functional pipeline which generates detail reports for each registrar_tld pair from Bigquery. The main features:
1. Deserialization from AVRO GenericRecord (from Bigquery) into BillingEvent, a POJO we control. This is especially valuable to enable intrinsic type-safety at the start of the pipeline.
2. Addition of .sql files containing the queries used to generate detail reports. These will later be templated to enable general usage.
3. Multi-file-writing within a single TextIO transform, which writes BillingEvents to different files based on their registrar_tld key combo.
This also upgrades the Beam core SDK referenced in repositories.bzl to 2.2.0 and returns the definitions to alphabetical order, to facilitate use of the check_bazel_deps.py script.
The final steps are:
- Converting this to a Nomulus command
- Templating the .sql queries
- @Injecting the @Config values for a given project
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178124838
