Update the Datastore to SQL migration pipeline (#927)

Browse source

* Update the Datastore to SQL migration pipeline

The pipeline now includes all entity types to be migrated by it, and has
completed successfully using the Sandbox data set. The running time in Sandbox
is about 3 hours, extrapolating by entity count to a 12-hour run with
production data. However, actual running time is likely to be longer since
throughput is lower with domains, which accounts for a higher percentage
of the total in production. More optimization will be needed.

The migrated data has not been validated.

...

This commit is contained in:

Weimin Yu 2021-01-12 18:05:46 -05:00 • committed by GitHub

parent 0109d5e473

commit 84d2b82050

No known key found for this signature in database

GPG key ID: 4AEE18F83AFDEB23

9 changed files with 1263 additions and 200 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1150

core/src/test/resources/google/registry/beam/initsql/pipeline_golden.dot

View file

File diff suppressed because it is too large Load diff

BIN

core/src/test/resources/google/registry/beam/initsql/pipeline_golden.png

View file

Binary file not shown.

Side by side Swipe Overlay Before Width:  |  Height:  |  Size: 929 KiB After Width:  |  Height:  |  Size: 1.1 MiB