Add a tools command to launch SQL validation job (#1526)

* Add a tools command to launch SQL validation job

Stopping using Pipeline.run().waitUntilFinish in
ValidateDatastorePipeline. Flex-templalate does not support blocking
wait in the main thread.

This PR adds a new ValidateSqlCommand that launches the pipeline and
maintains the SQL snapshot while the pipeline is running.

This PR also added more parameters to both ValidateSqlCommand and
ValidateDatastoreCommand:
- The -c option to supply an optional incremental comparison start time
- The -r option to supply an optional release tag that is not 'live',
  e.g., nomulus-DDDDYYMM-RC00

If the manual launch option (-m) is enabled, the commands will print the
gcloud command that can launch the pipeline.

Tested with sandbox, qa and the dev project.
This commit is contained in:
Weimin Yu 2022-02-28 13:14:57 -05:00 committed by GitHub
parent 55c368cf13
commit e47be4fa2c
17 changed files with 482 additions and 521 deletions

View file

@ -707,10 +707,7 @@ createToolTask(
'initSqlPipeline', 'google.registry.beam.initsql.InitSqlPipeline')
createToolTask(
'validateSqlPipeline', 'google.registry.beam.comparedb.ValidateSqlPipeline')
createToolTask(
'validateDatastorePipeline', 'google.registry.beam.comparedb.ValidateDatastorePipeline')
'validateDatabasePipeline', 'google.registry.beam.comparedb.ValidateDatabasePipeline')
createToolTask(
@ -797,15 +794,10 @@ if (environment == 'alpha') {
mainClass: 'google.registry.beam.rde.RdePipeline',
metaData : 'google/registry/beam/rde_pipeline_metadata.json'
],
validateDatastore :
validateDatabase :
[
mainClass: 'google.registry.beam.comparedb.ValidateDatastorePipeline',
metaData: 'google/registry/beam/validate_datastore_pipeline_metadata.json'
],
validateSql :
[
mainClass: 'google.registry.beam.comparedb.ValidateSqlPipeline',
metaData: 'google/registry/beam/validate_sql_pipeline_metadata.json'
mainClass: 'google.registry.beam.comparedb.ValidateDatabasePipeline',
metaData: 'google/registry/beam/validate_database_pipeline_metadata.json'
],
]
project.tasks.create("stageBeamPipelines") {