This makes a few cosmetic changes that prepares the pipeline for production.
Namely:
- Converts file names to include the input yearMonth, mostly mirroring the original invoicing pipeline.
- Factors out the yearMonth logic from the reporting module to the more common backend module. We will likely use the default yearMonth logic in other backend tasks (such as spec11 reporting).
- Adds the "withTemplateCompatability" flag to the Bigquery read, which allows multiple uses of the same template.
- Adds the 'billing' task queue, which retries up to 5 times every 3 minutes, which is about the rate we desire for checking if the pipeline is complete.
- Adds a shell 'invoicing upload' class, which tests the retry semantics we want for post-generation work (e-mailing the invoice to crr-tech, and publishing detail reports)
While this cl may look big, it's mostly just a refactor and setting up boilerplate needed to frame the upload logic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179849586
This establishes a fully functional pipeline which generates detail reports for each registrar_tld pair from Bigquery. The main features:
1. Deserialization from AVRO GenericRecord (from Bigquery) into BillingEvent, a POJO we control. This is especially valuable to enable intrinsic type-safety at the start of the pipeline.
2. Addition of .sql files containing the queries used to generate detail reports. These will later be templated to enable general usage.
3. Multi-file-writing within a single TextIO transform, which writes BillingEvents to different files based on their registrar_tld key combo.
This also upgrades the Beam core SDK referenced in repositories.bzl to 2.2.0 and returns the definitions to alphabetical order, to facilitate use of the check_bazel_deps.py script.
The final steps are:
- Converting this to a Nomulus command
- Templating the .sql queries
- @Injecting the @Config values for a given project
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178124838
There has been quite some descriptiveness between github and our internal build. I had to manually push a commit (1c1f95992a) to bring github up-to-date.
Now the github version is identical to what we'd get from doing a -dr-mkfoss. Hopefully the next time things will go smoothly.
The culprit turns out to be MOE itself. It was not attributing changes to commits correctly when the change involves moving files as a result of modifications made to moe_config.json. When moe_config.json is altered in a CL to move files around, MOE always thinks that move happens in the first commit to be pushed.
For example, if we have CL1,CL2,CL3, which correspond to CM1, CM2 and CM3 to be pushed to github, and a change to moe_config.json in CL3 moves a folder, MOE will think that move happens in CM1. This usually is not a problem when all commits are pushed, but when doing manual rebasing and cherry-picking, this will result in unintended dropped changes along with a commit.
I'll do a push to github early next week to confirm that things are back to normal.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177844920
This serves as proof-of-concept to verify we can use Beam for our invoice generation use case. Namely, it checks that we can:
- Deploy a Beam template to GCS
- Read from Bigquery within the template
- Run the template from App Engine
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175755390
This is the initial commit of the new billing system, rewritten as an Apache
Beam pipeline. This contains a basic end-to-end pipeline as proof of concept,
and boilerplate for GenerateInvoicesAction, which will eventually be our
automated invoice generation endpoint.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174184171
Now it lives alongside the delete prober data action, as well as any future
batch/maintenance tasks that should run periodically.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134435668
It is replaced by loadByForeignKey(), which does the same thing that
loadByUniqueId() did for contacts, hosts, and domains, and also
loadDomainApplication(), which loads domain application by ROID. This eliminates
the ugly mode-switching of attemping to load by other foreign key or ROID.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133980156
This is an internal-only feature that breaks the open source build.
CL created with:
dr-replace '(compatible_with.*)' '\1 # MOE:strip_line'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=128852873
