This adds actual subdomain verification via the SafeBrowsing API to the Spec11
pipeline, as well as on-the-fly KMS decryption via the GenerateSpec11Action to
securely store our API key in source code.
Testing the interaction becomes difficult due to serialization requirements, and will be significantly expanded in the next cl. For now, it verifies basic end-to-end pipeline behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208092942
It broke because I forgot to add the new spec11 packages to gtld.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206021827
This adds the scaffolding for a basic Spec11 pipeline- it gathers all domains from all time for a given project and counts how many there are. I've factored out a few common utilities for beam pipelines to avoid excessive duplication.
Future CLs will:
- Actually process domains via the SafeBrowsing API
- Generate a real spec11 report
- Template queries based on the input YearMonth
- Abstract more commonalities across beam pipelines to reduce boilerplate when adding new pipelines.
TESTED: FOSS test passed, and ran successfully on alpha
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205997741
This prepares for the spec11 beam pipeline to live parallel to the invoicing
beam pipeline, for better organization.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204980582
This officially adds a 15% discount to sunrise creates and makes anchor tenant
creates free for the first 2 years.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204805141
This removes the following unnecessary imports:
//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
This moves the new pipeline's invoice generation to the billing bucket, under the 'invoices/yyyy-MM' subdirectory.
This also changes the invoice e-mail to use a multipart message that attaches the invoice to the e-mail, to guarantee the correct MIME type and download.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181746191
This closes the end-to-end billing pipeline, allowing us to share generated detail reports with registrars via Drive and e-mail the invoicing team a link to the generated invoice.
This also factors out the email configs from ICANN reporting into the common 'misc' config, since we'll likely need alert e-mails for future periodic tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180805972
This makes a few cosmetic changes that prepares the pipeline for production.
Namely:
- Converts file names to include the input yearMonth, mostly mirroring the original invoicing pipeline.
- Factors out the yearMonth logic from the reporting module to the more common backend module. We will likely use the default yearMonth logic in other backend tasks (such as spec11 reporting).
- Adds the "withTemplateCompatability" flag to the Bigquery read, which allows multiple uses of the same template.
- Adds the 'billing' task queue, which retries up to 5 times every 3 minutes, which is about the rate we desire for checking if the pipeline is complete.
- Adds a shell 'invoicing upload' class, which tests the retry semantics we want for post-generation work (e-mailing the invoice to crr-tech, and publishing detail reports)
While this cl may look big, it's mostly just a refactor and setting up boilerplate needed to frame the upload logic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179849586
This creates an end-to-end test that checks for proper billing pipeline IO writes. The only remaining test would be to add a test for the Bigquery query, but see b/70839142 for why I've deemed that more work than worthwhile.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179706561
