This CL also fixes a bug. Registrars were returned in an arbitrary order. This caused cursor-based pagination to fail. Now we always sort by registrar name (even for handle searches), and use the registrar name in the cursor, to ensure proper behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182098187
The new version of the framework that works correctly for this isn't
externally available quite yet, so leave it at text/plain for the time
being.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181817266
This moves the new pipeline's invoice generation to the billing bucket, under the 'invoices/yyyy-MM' subdirectory.
This also changes the invoice e-mail to use a multipart message that attaches the invoice to the e-mail, to guarantee the correct MIME type and download.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181746191
Next up is adding custom logic so that the results of these checks can be
more meaningful.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181660956
By default, RDAP entity searches return both contacts and registrars. This CL
adds a new query parameter to request only one or the other. Among other
benefits, this will allow a future CL to permit wildcard searches that return
all registrars.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181605990
The next step is to add them for domain checks as well (which is simpler
because it doesn't involve validation).
This requires the addition of a TrimWhitespaceAdapter for XML JAXB objects,
which will prove useful for other @XmlValue attributes in the future.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181526726
This creates a specified number of tokens of a given schema, with a dryrun option
to not persist them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181403775
The extension itself isn't used for anything yet; allocation tokens can be
passed for domain creates and checks but are ignored if present. This will
be changed in a subsequent CL that adds AllocationToken entities and related
logic. Usage of this extension in any other EPP flow will throw an
UnsupportedExtensionException.
The relevant spec is https://tools.ietf.org/html/draft-gould-allocation-token-04
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181343188
Logic actually using this entity to follow in subsequent CLs introducing
the command to generate/save these entities as well as the flow logic for
considering them during domain EPP operations.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181342600
This closes the end-to-end billing pipeline, allowing us to share generated detail reports with registrars via Drive and e-mail the invoicing team a link to the generated invoice.
This also factors out the email configs from ICANN reporting into the common 'misc' config, since we'll likely need alert e-mails for future periodic tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180805972
This makes a few cosmetic changes that prepares the pipeline for production.
Namely:
- Converts file names to include the input yearMonth, mostly mirroring the original invoicing pipeline.
- Factors out the yearMonth logic from the reporting module to the more common backend module. We will likely use the default yearMonth logic in other backend tasks (such as spec11 reporting).
- Adds the "withTemplateCompatability" flag to the Bigquery read, which allows multiple uses of the same template.
- Adds the 'billing' task queue, which retries up to 5 times every 3 minutes, which is about the rate we desire for checking if the pipeline is complete.
- Adds a shell 'invoicing upload' class, which tests the retry semantics we want for post-generation work (e-mailing the invoice to crr-tech, and publishing detail reports)
While this cl may look big, it's mostly just a refactor and setting up boilerplate needed to frame the upload logic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179849586
ArrayList is more performant and there's no reason to use a LinkedList here.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179717525
Stream.concat only accepts 2 parameters. Streams.concat on the other hand
accepts any number of parameters.
Moving to Streams.concat for all uses (2 or more) makes sense for uniformity
and convenience reasons.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179716648
This creates an end-to-end test that checks for proper billing pipeline IO writes. The only remaining test would be to add a test for the Bigquery query, but see b/70839142 for why I've deemed that more work than worthwhile.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179706561
In Truth8, we can do assertThat(stream) directly. It's less verbose and clearer
in most cases.
Note that for the "finishers" (e.g. "containsExactyElementsIn") - streams are
still not allowed. So when there is:
assertThat(stream.map(someTransformation).collect(toList()))
.containsExactlyElementsIn(expecteStream.map(someTransformation).collect(toList()));
I kept the .collect in the assertThat to preserve the symmetry with the
finisher.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179697587
This also incorporates general improvements and additions to the existing EPP
lifecycle tests around domain deletion. As a refresher: There is a 5 day
add grace period (AGP) following domain creation. Domains that are deleted
during that period have their create costs (but not EAP costs) refunded. This
deletion takes place immediately. Refunds are implemented by issuing a
Cancellation for the associated OneTime billing event.
Domains that are deleted after AGP ends first go through a 30 day redemption
grace period followed by a 5 day pending deletion period. No create fees are
refunded in this case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179597874
This is done first and formost to stop "empty" commits that cause errors in
publishDnsUpdates. The reason being that the Cloud DNS api fails when there are
no updates at all in a change.
Allowing this is a requirement for the writer to be idempotent - if we delete a
domain, then run the writer to delete it again - we'll get 0 additions and 0
deletions which fails.
This isn't theoretical either - we've seen it happen, causing a
publishDnsUpdates to fail over and over again.
While fixing this, we also remove all RRS that are common between additions and
deletions. This is just an optimization and shouldn't affect behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179525218
In addition, while adding the tests, I became discontented with the thoroughness of the cursor navigation tests, which checked only the number of items returned, not their proper ordering. So I updated them to be more careful, and backported the changes to the nameserver and entity search tests as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179442118
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238852
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238745
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238504
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179221800
These checks were removed in [] and re-adding them is the last
step of the migration to using expectThrows/assertThrows globally.
Note that this is roughly half of them. More to come in a follow-up CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179216707
The assertThrows/expectThrows refactoring script does not use method
references, apparently.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089048
A couple methods were moved to new locations so they are accessible to all types of search queries, not just nameservers like they originally were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089014
This is needed to fix an inability in Java 8 to correctly infer the type
when the transaction was being allowed to return the value it loaded. The
error was:
INFO: Compilation unit has error diagnostics: [third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:109: error: incompatible types: inference variable R has incompatible bounds
ofy().transact(() -> rdeImportUtils.importEppResource(newContact));
^
upper bounds: java.lang.Object
lower bounds: void, third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:132:
error: incompatible types: inference variable R has incompatible bounds
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179082154
we set the "denial of existence" to NSEC (rather than NSEC3), because preventing "walking the zone" isn't an issue for TLDs.
It uses the default security configuration for everything else, which at the time of this writing is:
Key signing: RSASHA256, key length of 2048
Zone signing: RSASHA256, key length of 1024
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179045575
It was easier to simply move these over manually than to try to debug
the automated tooling.
I also changed the case in an exception message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178926365
These testing helper functions can't be handled by the automatic refactoring
tool because they're taking in expected exception details as parameters.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178832406
This is in preparation for running the automatic refactoring script that
will replace all ExpectedExceptions with use of JUnit 4.13's assertThrows/
expectThrows.
Note that I have recorded the callsites of assertions about EppExceptions
being marshallable and will edit those specific assertions back in after
running the automatic refactoring script (which do not understand these).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178812403
The quotas can be configured in the yaml configuration file. Default quota will be applied to any userId that is not matched in the custom quota list.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178804649
Detailed discussion can be found on github (https://github.com/bazelbuild/bazel/issues/3215). On newer Debian systems as well as on macOS, the temporary folder created to store gpg configs are too long, which results in an error:
gpg: can't connect to the agent: File name too long
This CL makes affected tests use /tmp as temp folder base. Other options are discussed in the github issue, but this one seems to be the mostly non-harmful one. Note that this change only affect the FOSS build, as internal builds are not affected by the temp directory issue, probably due to the internal build tool using a different base temp directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178803914
This fixes a bug where collections of incompatible types are being tested for equality to each other (e.g.: Set<Foo> equals Set<Integer> should never return true unless both sets are empty, a bit of a vacuous assertion).
This change is necessary to unblock future improvements to the static analysis capabilities of the java compiler.
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178798071
It turns out that the RDAP spec does not envision multiple help pages. We can
still support them (for the TOS, for instance), but we shouldn't expect users
to go searching for help other than the main page. Therefore, consolidate the
useful information on the main page, and get rid of some of the others.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178792548
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.
Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
