Add preliminary spec11 monthly pipeline

This adds the scaffolding for a basic Spec11 pipeline- it gathers all domains from all time for a given project and counts how many there are. I've factored out a few common utilities for beam pipelines to avoid excessive duplication.

Future CLs will:
- Actually process domains via the SafeBrowsing API
- Generate a real spec11 report
- Template queries based on the input YearMonth
- Abstract more commonalities across beam pipelines to reduce boilerplate when adding new pipelines.

TESTED: FOSS test passed, and ran successfully on alpha

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205997741
This commit is contained in:
larryruili 2018-07-25 08:34:58 -07:00 committed by jianglai
parent ded40851d3
commit d199b383e5
14 changed files with 252 additions and 38 deletions

View file

@ -520,25 +520,38 @@ public final class RegistryConfig {
}
/**
* Returns the URL of the GCS location we store jar dependencies for the invoicing pipeline.
* Returns the URL of the GCS location for storing the monthly spec11 Beam template.
*
* @see google.registry.beam.invoicing.InvoicingPipeline
* @see google.registry.beam.spec11.Spec11Pipeline
*/
@Provides
@Config("invoiceStagingUrl")
@Config("spec11TemplateUrl")
public static String provideSpec11TemplateUrl(
@Config("apacheBeamBucketUrl") String beamBucketUrl) {
return beamBucketUrl + "/templates/spec11";
}
/**
* Returns the URL of the GCS location we store jar dependencies for beam pipelines.
*
* @see google.registry.beam.invoicing.InvoicingPipeline
* @see google.registry.beam.spec11.Spec11Pipeline
*/
@Provides
@Config("beamStagingUrl")
public static String provideInvoiceStagingUrl(
@Config("apacheBeamBucketUrl") String beamBucketUrl) {
return beamBucketUrl + "/staging";
}
/**
* Returns the Google Cloud Storage bucket for ICANN transaction and activity reports to
* be uploaded.
* Returns the Google Cloud Storage bucket for Spec11 and ICANN transaction and activity reports
* to be uploaded.
*
* @see google.registry.reporting.icann.IcannReportingUploadAction
*/
@Provides
@Config("icannReportingBucket")
@Config("reportingBucket")
public static String provideIcannReportingBucket(@Config("projectId") String projectId) {
return projectId + "-reporting";
}
@ -588,6 +601,17 @@ public final class RegistryConfig {
return "gs://" + billingBucket;
}
/**
* Returns the URL of the GCS subdirectory we store Spec11 reports in.
*
* @see google.registry.beam.spec11.Spec11Pipeline
*/
@Provides
@Config("spec11BucketUrl")
public static String provideSpec11BucketUrl(@Config("reportingBucket") String reportingBucket) {
return "gs://" + reportingBucket + "/icann/spec11";
}
/**
* Returns whether or not we should publish invoices to partners automatically by default.
*