*** Reason for rollback ***
Breaks the FOSS build.
We'll reincorporate this change once Closure Rules is properly updated to accommodate it.
*** Original change description ***
Change all foreach loops in Soy templates to use the for loop syntax
Soy supports 2 kinds of loops:
foreach- for iterating over items in a collection e.g. {foreach $item in $list}...{/foreach}
for - for indexed iteration e.g. {for $i in range(0, 10)}...{/for}
The reason Soy has 2 different loops is an accident of history, Soy didn’t use to have a proper grammar for expressions and so the alternate ‘for...range’ syntax was added to make it possible to write indexed loops. As the gramma...
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180961695
Soy supports 2 kinds of loops:
foreach- for iterating over items in a collection e.g. {foreach $item in $list}...{/foreach}
for - for indexed iteration e.g. {for $i in range(0, 10)}...{/for}
The reason Soy has 2 different loops is an accident of history, Soy didn’t use to have a proper grammar for expressions and so the alternate ‘for...range’ syntax was added to make it possible to write indexed loops. As the grammar has improved having the two syntaxes is no longer necessary and so we are eliminating one of them.
As of 4a7373333f or mvn release "2018-01-03" the two forms are actually aliases for one another, so the only difference is the keyword (‘for’ vs ‘foreach’), and while the foreach loop is more popular the ‘for’ terminology is more standard so we are switching everything to that.
LSC: []
Tested:
TAP sample presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180942763
This closes the end-to-end billing pipeline, allowing us to share generated detail reports with registrars via Drive and e-mail the invoicing team a link to the generated invoice.
This also factors out the email configs from ICANN reporting into the common 'misc' config, since we'll likely need alert e-mails for future periodic tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180805972
Soy is going to disallow passing params that are unused in the called template and its dependencies. This CL removes these unused params from the call sites.
Passing an unused param might indicate a bug such as having a typo in an optional parameter. Please review this CL carefully and edit the code in Critique if this is the case.
More information: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180571605
This makes a few cosmetic changes that prepares the pipeline for production.
Namely:
- Converts file names to include the input yearMonth, mostly mirroring the original invoicing pipeline.
- Factors out the yearMonth logic from the reporting module to the more common backend module. We will likely use the default yearMonth logic in other backend tasks (such as spec11 reporting).
- Adds the "withTemplateCompatability" flag to the Bigquery read, which allows multiple uses of the same template.
- Adds the 'billing' task queue, which retries up to 5 times every 3 minutes, which is about the rate we desire for checking if the pipeline is complete.
- Adds a shell 'invoicing upload' class, which tests the retry semantics we want for post-generation work (e-mailing the invoice to crr-tech, and publishing detail reports)
While this cl may look big, it's mostly just a refactor and setting up boilerplate needed to frame the upload logic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179849586
Stream.concat only accepts 2 parameters. Streams.concat on the other hand
accepts any number of parameters.
Moving to Streams.concat for all uses (2 or more) makes sense for uniformity
and convenience reasons.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179716648
This creates an end-to-end test that checks for proper billing pipeline IO writes. The only remaining test would be to add a test for the Bigquery query, but see b/70839142 for why I've deemed that more work than worthwhile.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179706561
This also incorporates general improvements and additions to the existing EPP
lifecycle tests around domain deletion. As a refresher: There is a 5 day
add grace period (AGP) following domain creation. Domains that are deleted
during that period have their create costs (but not EAP costs) refunded. This
deletion takes place immediately. Refunds are implemented by issuing a
Cancellation for the associated OneTime billing event.
Domains that are deleted after AGP ends first go through a 30 day redemption
grace period followed by a 5 day pending deletion period. No create fees are
refunded in this case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179597874
This is done first and formost to stop "empty" commits that cause errors in
publishDnsUpdates. The reason being that the Cloud DNS api fails when there are
no updates at all in a change.
Allowing this is a requirement for the writer to be idempotent - if we delete a
domain, then run the writer to delete it again - we'll get 0 additions and 0
deletions which fails.
This isn't theoretical either - we've seen it happen, causing a
publishDnsUpdates to fail over and over again.
While fixing this, we also remove all RRS that are common between additions and
deletions. This is just an optimization and shouldn't affect behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179525218
In addition, while adding the tests, I became discontented with the thoroughness of the cursor navigation tests, which checked only the number of items returned, not their proper ordering. So I updated them to be more careful, and backported the changes to the nameserver and entity search tests as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179442118
That's 50 each for frontend and backend and 5 for tools. Since the
MetricExporter bug has been fixed for awhile now, we aren't gaining anything by
artificially keeping the instance number low, whereas we might benefit from
higher instance counts, e.g. for load-testing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179432038
Right now VoidDnsWriter is injected in the tools - meaning it's possible to
*set* VoidDnsWriter as the writer of a TLD - but it isn't injected in the
backend - meaning we get an error if we actually try to use it.
We need VoidDnsWriter at least for load-testing, and in general for any test
TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179425574
A couple methods were moved to new locations so they are accessible to all types of search queries, not just nameservers like they originally were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089014
This is needed to fix an inability in Java 8 to correctly infer the type
when the transaction was being allowed to return the value it loaded. The
error was:
INFO: Compilation unit has error diagnostics: [third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:109: error: incompatible types: inference variable R has incompatible bounds
ofy().transact(() -> rdeImportUtils.importEppResource(newContact));
^
upper bounds: java.lang.Object
lower bounds: void, third_party/java_src/gtld/javatests/google/registry/rde/imports/RdeImportUtilsTest.java:132:
error: incompatible types: inference variable R has incompatible bounds
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179082154
we set the "denial of existence" to NSEC (rather than NSEC3), because preventing "walking the zone" isn't an issue for TLDs.
It uses the default security configuration for everything else, which at the time of this writing is:
Key signing: RSASHA256, key length of 2048
Zone signing: RSASHA256, key length of 1024
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179045575
It was easier to simply move these over manually than to try to debug
the automated tooling.
I also changed the case in an exception message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178926365
It's untested, we suspect it's not actually working properly, and we don't
intend to ever need to use anything having to do with auctions ever again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178815580
The quotas can be configured in the yaml configuration file. Default quota will be applied to any userId that is not matched in the custom quota list.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178804649
This forks off the results of the billing_events query from Bigquery and generates the overall invoice for the month.
This also turns the ad hoc beam binary into a proper Nomulus command, allowing us to @Inject from the RegsistryToolComponent and thus template the BillingEvent Bigquery query.
Next steps:
- Add the beam package to the open source build
- That will likely require updates to the open source jars, since I'm using features from 2.2.0 which was only just released.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178798943
It turns out that the RDAP spec does not envision multiple help pages. We can
still support them (for the TOS, for instance), but we shouldn't expect users
to go searching for help other than the main page. Therefore, consolidate the
useful information on the main page, and get rid of some of the others.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178792548
This also cleans up a few miscellaneous code quality issues encountered
while adding the new setter: using a cleaner way to conditionally set field
values, documenting the format of the add grace period parameters, and
improves some code comments and formatting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178387731
Before pushing an update to Cloud DNS, the CloudDnsWriter needs to read all the domain RRSs from Cloud DNS one by one to know what to delete.
Doing so sequentially results in update times that are too long (approx 200ms per domain, which is 20 seconds per batch of 100) severely limiting our QPS.
This CL uses Concurrent threading to do the Cloud DNS queries in parallel. Unfortunately, my preferred method (Set.parallelStream) doesn't work on App Engine :(
This reduces the per-item time from 200ms to 80ms, which can be further reduced to 50ms if we remove the rate limiter (currently set to 20 per second).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178126877
This establishes a fully functional pipeline which generates detail reports for each registrar_tld pair from Bigquery. The main features:
1. Deserialization from AVRO GenericRecord (from Bigquery) into BillingEvent, a POJO we control. This is especially valuable to enable intrinsic type-safety at the start of the pipeline.
2. Addition of .sql files containing the queries used to generate detail reports. These will later be templated to enable general usage.
3. Multi-file-writing within a single TextIO transform, which writes BillingEvents to different files based on their registrar_tld key combo.
This also upgrades the Beam core SDK referenced in repositories.bzl to 2.2.0 and returns the definitions to alphabetical order, to facilitate use of the check_bazel_deps.py script.
The final steps are:
- Converting this to a Nomulus command
- Templating the .sql queries
- @Injecting the @Config values for a given project
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178124838
This solves the problem of external poll message IDs not being globally
unique by simply appending the event year. This means that autorenew poll
messages will increment by one every year, so they will always be unique.
This also requires no data schema changes, and thus most importantly, no
data migration.
Incoming requests lacking this new year field will continue to work for
now for backwards compatibility reasons. This is possible because we don't
actually use the year for anything.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178012685
Currently, if for some reason publishDnsUpdates gets a request to publish
domains to a DnsWriter that doesn't belong to said domain - it logs a warning
but published anyway.
This can happen when Writers are changed (swapped for a different writer)
leaving update commands "stuck" with the wrong writer.
Normally you'd expect these update commands to just publish their data and be
on their way. However, if the update fails for some reason (likely - if the
Writer change happened BECAUSE the updates are failing) then the same
publishDnsUpdate command will continue to run forever.
This CL changes the behavior for "publish to wrong DnsWriter" to instead
requeue the batched domains / hosts back to the Dns-pull queue, allowing them
to be re-batched (and hence published) with the correct DnsWriter(s). This
re-batching will take place in ReadDnsQueueAction.java
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177863076
There has been quite some descriptiveness between github and our internal build. I had to manually push a commit (1c1f95992a) to bring github up-to-date.
Now the github version is identical to what we'd get from doing a -dr-mkfoss. Hopefully the next time things will go smoothly.
The culprit turns out to be MOE itself. It was not attributing changes to commits correctly when the change involves moving files as a result of modifications made to moe_config.json. When moe_config.json is altered in a CL to move files around, MOE always thinks that move happens in the first commit to be pushed.
For example, if we have CL1,CL2,CL3, which correspond to CM1, CM2 and CM3 to be pushed to github, and a change to moe_config.json in CL3 moves a folder, MOE will think that move happens in CM1. This usually is not a problem when all commits are pushed, but when doing manual rebasing and cherry-picking, this will result in unintended dropped changes along with a commit.
I'll do a push to github early next week to confirm that things are back to normal.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177844920
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
Failing to use the fee extension during EAP can result in charges to registrars
that are radically different than what they may have been expecting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177597883
This also improves performance by calculating a set once rather than
on-demand every time it is used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177471572
Loadtesting data is identified as "prober data" by this job (it removes
anything under ".test", not only prober data)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177309096
DummyKeyringModule provides a fake string as the JSON credential used to instantiate a GoogleCredential. Of course this would not work and when the metric reporter requests a GoogleCredential in the main thread. This causes the FOSS build to crash on startup, because it defaults to use DummyKeyringModule.
This change allows a graceful handling of such an error by wrapping any calls to instantiate a metric reporter in a try block. Note that any attempt to write to stackdriver will still fail, but that happens in a different thread and will not make the whole program crash.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177183337
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
*** Reason for rollback ***
Going with a safer approach to using fresh poll message IDs that doesn't mutate domains themselves.
*** Original change description ***
Use PollMessage IDs that are globally unique across all time
The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177081870
Also, a couple tests are reworked to make their output more understandable in
case of failure.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
This command is used by registry operators to apply registry locks to
domain names.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176577240
