The new version of the framework that works correctly for this isn't
externally available quite yet, so leave it at text/plain for the time
being.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181817266
This moves the new pipeline's invoice generation to the billing bucket, under the 'invoices/yyyy-MM' subdirectory.
This also changes the invoice e-mail to use a multipart message that attaches the invoice to the e-mail, to guarantee the correct MIME type and download.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181746191
Next up is adding custom logic so that the results of these checks can be
more meaningful.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181660956
By default, RDAP entity searches return both contacts and registrars. This CL
adds a new query parameter to request only one or the other. Among other
benefits, this will allow a future CL to permit wildcard searches that return
all registrars.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181605990
The next step is to add them for domain checks as well (which is simpler
because it doesn't involve validation).
This requires the addition of a TrimWhitespaceAdapter for XML JAXB objects,
which will prove useful for other @XmlValue attributes in the future.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181526726
This creates a specified number of tokens of a given schema, with a dryrun option
to not persist them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181403775
The extension itself isn't used for anything yet; allocation tokens can be
passed for domain creates and checks but are ignored if present. This will
be changed in a subsequent CL that adds AllocationToken entities and related
logic. Usage of this extension in any other EPP flow will throw an
UnsupportedExtensionException.
The relevant spec is https://tools.ietf.org/html/draft-gould-allocation-token-04
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181343188
Logic actually using this entity to follow in subsequent CLs introducing
the command to generate/save these entities as well as the flow logic for
considering them during domain EPP operations.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181342600
This is a temporary change used for b/71607306 only, and has TODOs to revert at
every change once the bug is done.
We want to check how Cloud DNS handles large (1M+ domain) zones, especially
during resigning. However, due to a separate bug (b/70980350, and maybe another
one) we can't currently create such a large zone in nomulus within the required
4 day timeframe. The most we managed is 300k domains.
We could wait until the bug is fixed, but if there's a problem with Cloud DNS -
we want to find out as fast as possible. Hence, this CL that allows us to
register 1M domains by creating "just" 100k domains in nomulus.
The CL creates a new "MultiplyingCloudDnsWriter" writer, that when publishing a
domain, pretends that we are publishing 10 domains (with 9 additional
"fictional" domains, that get their Datastore data from the actual domain).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180962149
*** Reason for rollback ***
Breaks the FOSS build.
We'll reincorporate this change once Closure Rules is properly updated to accommodate it.
*** Original change description ***
Change all foreach loops in Soy templates to use the for loop syntax
Soy supports 2 kinds of loops:
foreach- for iterating over items in a collection e.g. {foreach $item in $list}...{/foreach}
for - for indexed iteration e.g. {for $i in range(0, 10)}...{/for}
The reason Soy has 2 different loops is an accident of history, Soy didn’t use to have a proper grammar for expressions and so the alternate ‘for...range’ syntax was added to make it possible to write indexed loops. As the gramma...
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180961695
Soy supports 2 kinds of loops:
foreach- for iterating over items in a collection e.g. {foreach $item in $list}...{/foreach}
for - for indexed iteration e.g. {for $i in range(0, 10)}...{/for}
The reason Soy has 2 different loops is an accident of history, Soy didn’t use to have a proper grammar for expressions and so the alternate ‘for...range’ syntax was added to make it possible to write indexed loops. As the grammar has improved having the two syntaxes is no longer necessary and so we are eliminating one of them.
As of 4a7373333f or mvn release "2018-01-03" the two forms are actually aliases for one another, so the only difference is the keyword (‘for’ vs ‘foreach’), and while the foreach loop is more popular the ‘for’ terminology is more standard so we are switching everything to that.
LSC: []
Tested:
TAP sample presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180942763
This closes the end-to-end billing pipeline, allowing us to share generated detail reports with registrars via Drive and e-mail the invoicing team a link to the generated invoice.
This also factors out the email configs from ICANN reporting into the common 'misc' config, since we'll likely need alert e-mails for future periodic tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180805972
Soy is going to disallow passing params that are unused in the called template and its dependencies. This CL removes these unused params from the call sites.
Passing an unused param might indicate a bug such as having a typo in an optional parameter. Please review this CL carefully and edit the code in Critique if this is the case.
More information: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180571605
This makes a few cosmetic changes that prepares the pipeline for production.
Namely:
- Converts file names to include the input yearMonth, mostly mirroring the original invoicing pipeline.
- Factors out the yearMonth logic from the reporting module to the more common backend module. We will likely use the default yearMonth logic in other backend tasks (such as spec11 reporting).
- Adds the "withTemplateCompatability" flag to the Bigquery read, which allows multiple uses of the same template.
- Adds the 'billing' task queue, which retries up to 5 times every 3 minutes, which is about the rate we desire for checking if the pipeline is complete.
- Adds a shell 'invoicing upload' class, which tests the retry semantics we want for post-generation work (e-mailing the invoice to crr-tech, and publishing detail reports)
While this cl may look big, it's mostly just a refactor and setting up boilerplate needed to frame the upload logic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179849586
ArrayList is more performant and there's no reason to use a LinkedList here.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179717525
Stream.concat only accepts 2 parameters. Streams.concat on the other hand
accepts any number of parameters.
Moving to Streams.concat for all uses (2 or more) makes sense for uniformity
and convenience reasons.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179716648
This creates an end-to-end test that checks for proper billing pipeline IO writes. The only remaining test would be to add a test for the Bigquery query, but see b/70839142 for why I've deemed that more work than worthwhile.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179706561
In Truth8, we can do assertThat(stream) directly. It's less verbose and clearer
in most cases.
Note that for the "finishers" (e.g. "containsExactyElementsIn") - streams are
still not allowed. So when there is:
assertThat(stream.map(someTransformation).collect(toList()))
.containsExactlyElementsIn(expecteStream.map(someTransformation).collect(toList()));
I kept the .collect in the assertThat to preserve the symmetry with the
finisher.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179697587
This also incorporates general improvements and additions to the existing EPP
lifecycle tests around domain deletion. As a refresher: There is a 5 day
add grace period (AGP) following domain creation. Domains that are deleted
during that period have their create costs (but not EAP costs) refunded. This
deletion takes place immediately. Refunds are implemented by issuing a
Cancellation for the associated OneTime billing event.
Domains that are deleted after AGP ends first go through a 30 day redemption
grace period followed by a 5 day pending deletion period. No create fees are
refunded in this case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179597874
This is done first and formost to stop "empty" commits that cause errors in
publishDnsUpdates. The reason being that the Cloud DNS api fails when there are
no updates at all in a change.
Allowing this is a requirement for the writer to be idempotent - if we delete a
domain, then run the writer to delete it again - we'll get 0 additions and 0
deletions which fails.
This isn't theoretical either - we've seen it happen, causing a
publishDnsUpdates to fail over and over again.
While fixing this, we also remove all RRS that are common between additions and
deletions. This is just an optimization and shouldn't affect behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179525218
In addition, while adding the tests, I became discontented with the thoroughness of the cursor navigation tests, which checked only the number of items returned, not their proper ordering. So I updated them to be more careful, and backported the changes to the nameserver and entity search tests as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179442118
That's 50 each for frontend and backend and 5 for tools. Since the
MetricExporter bug has been fixed for awhile now, we aren't gaining anything by
artificially keeping the instance number low, whereas we might benefit from
higher instance counts, e.g. for load-testing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179432038
Right now VoidDnsWriter is injected in the tools - meaning it's possible to
*set* VoidDnsWriter as the writer of a TLD - but it isn't injected in the
backend - meaning we get an error if we actually try to use it.
We need VoidDnsWriter at least for load-testing, and in general for any test
TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179425574
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238852
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238745
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179238504
Apologies for the reformatting, but this refactoring is quite rote and it's
definitely a bigger use of total time to perform the reformatting individually
than to simply do it file-wide.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179221800
These checks were removed in [] and re-adding them is the last
step of the migration to using expectThrows/assertThrows globally.
Note that this is roughly half of them. More to come in a follow-up CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179216707
The assertThrows/expectThrows refactoring script does not use method
references, apparently.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089048
