mirror of
https://github.com/google/nomulus.git
synced 2025-08-02 07:52:11 +02:00
Remove datastore related code (#1906)
This commit is contained in:
parent
913edb23ee
commit
e41fd7877e
152 changed files with 886 additions and 4460 deletions
|
@ -99,5 +99,5 @@ that are used by a large number of other tools are:
|
|||
* **`ListObjectsCommand`** -- Lists all objects of a specific type from the
|
||||
server and outputs them.
|
||||
* **`MutatingCommand`** -- Provides a facility to create or update entities in
|
||||
Datastore, and uses a diff algorithm to display the changes that will be
|
||||
the database, and uses a diff algorithm to display the changes that will be
|
||||
made before committing them.
|
||||
|
|
|
@ -120,17 +120,6 @@ specification for the cron task.
|
|||
Here are the task queues in use by the system. All are push queues unless
|
||||
explicitly marked as otherwise.
|
||||
|
||||
* `async-delete-pull` and `async-host-rename-pull` -- Pull queues for tasks to
|
||||
asynchronously delete contacts/hosts and to asynchronously refresh DNS for
|
||||
renamed hosts, respectively. Tasks are enqueued during EPP flows and then
|
||||
handled in batches by the regularly running cron tasks
|
||||
`DeleteContactsAndHostsAction` and `RefreshDnsOnHostRenameAction`.
|
||||
* `bigquery-streaming-metrics` -- Queue for metrics that are asynchronously
|
||||
streamed to BigQuery in the `Metrics` class. Tasks are enqueued during EPP
|
||||
flows in `EppController`. This means that there is a lag of a few seconds to
|
||||
a few minutes between when metrics are generated and when they are queryable
|
||||
in BigQuery, but this is preferable to slowing all EPP flows down and
|
||||
blocking them on BigQuery streaming.
|
||||
* `brda` -- Queue for tasks to upload weekly Bulk Registration Data Access
|
||||
(BRDA) files to a location where they are available to ICANN. The
|
||||
`RdeStagingReducer` (part of the RDE MapReduce) creates these tasks at the
|
||||
|
@ -141,33 +130,6 @@ explicitly marked as otherwise.
|
|||
`DnsWriter` for the TLD.
|
||||
* `dns-publish` -- Queue for batches of DNS updates to be pushed to DNS
|
||||
writers.
|
||||
* `export-bigquery-poll` -- Queue for tasks to query the success/failure of a
|
||||
given BigQuery export job. Tasks are enqueued by `BigqueryPollJobAction`.
|
||||
* `export-commits` -- Queue for tasks to export commit log checkpoints. Tasks
|
||||
are enqueued by `CommitLogCheckpointAction` (which is run every minute by
|
||||
cron) and executed by `ExportCommitLogDiffAction`.
|
||||
* `export-snapshot` -- Cron and push queue for tasks to load a Datastore
|
||||
snapshot that was stored in Google Cloud Storage and export it to BigQuery.
|
||||
Tasks are enqueued by both cron and `CheckSnapshotAction` and are executed
|
||||
by both `ExportSnapshotAction` and `LoadSnapshotAction`.
|
||||
* `export-snapshot-poll` -- Queue for tasks to check that a Datastore snapshot
|
||||
has been successfully uploaded to Google Cloud Storage (this is an
|
||||
asynchronous background operation that can take an indeterminate amount of
|
||||
time). Once the snapshot is successfully uploaded, it is imported into
|
||||
BigQuery. Tasks are enqueued by `ExportSnapshotAction` and executed by
|
||||
`CheckSnapshotAction`.
|
||||
* `export-snapshot-update-view` -- Queue for tasks to update the BigQuery
|
||||
views to point to the most recently uploaded snapshot. Tasks are enqueued by
|
||||
`LoadSnapshotAction` and executed by `UpdateSnapshotViewAction`.
|
||||
* `group-members-sync` -- Cron queue for tasks to sync registrar contacts (not
|
||||
domain contacts!) to Google Groups. Tasks are executed by
|
||||
`SyncGroupMembersAction`.
|
||||
* `load[0-9]` -- Queues used to load-test the system by `LoadTestAction`.
|
||||
These queues don't need to exist except when actively running load tests
|
||||
(running load tests on production environments is not recommended). There
|
||||
are ten of these queues to provide simple sharding, because Nomulus is
|
||||
capable of handling significantly more Queries Per Second than the highest
|
||||
throttle limit available on task queues (which is 500 qps).
|
||||
* `lordn-claims` and `lordn-sunrise` -- Pull queues for handling LORDN
|
||||
exports. Tasks are enqueued synchronously during EPP commands depending on
|
||||
whether the domain name in question has a claims notice ID.
|
||||
|
@ -298,109 +260,19 @@ of experience running a production registry using this codebase.
|
|||
errors, it can be pushed to Production.
|
||||
5. Repeat once weekly, or potentially more often.
|
||||
|
||||
## Cloud Datastore
|
||||
## Cloud SQL
|
||||
|
||||
Nomulus uses [Cloud
|
||||
Datastore](https://cloud.google.com/appengine/docs/java/datastore/) as its
|
||||
primary database. Cloud Datastore is a NoSQL document database that provides
|
||||
automatic horizontal scaling, high performance, and high availability. All
|
||||
information that is persisted to Cloud Datastore takes the form of Java classes
|
||||
annotated with `@Entity` that are located in the `model` package. The [Objectify
|
||||
library](https://cloud.google.com/appengine/docs/java/gettingstarted/using-datastore-objectify)
|
||||
is used to persist instances of these classes in a format that Datastore
|
||||
understands.
|
||||
|
||||
A brief overview of the different entity types found in the App Engine Datastore
|
||||
Viewer may help administrators understand what they are seeing. Note that some
|
||||
of these entities are part of App Engine tools that are outside of the domain
|
||||
registry codebase:
|
||||
|
||||
* `_AE_*` -- These entities are created by App Engine.
|
||||
* `_ah_SESSION` -- These entities track App Engine client sessions.
|
||||
* `_GAE_MR_*` -- These entities are generated by App Engine while running
|
||||
MapReduces.
|
||||
* `BackupStatus` -- There should only be one of these entities, used to
|
||||
maintain the state of the backup process.
|
||||
* `Cancellation` -- A cancellation is a special type of billing event which
|
||||
represents the cancellation of another billing event such as a OneTime or
|
||||
Recurring.
|
||||
* `ClaimsList`, `ClaimsListShard`, and `ClaimsListSingleton` -- These entities
|
||||
store the TMCH claims list, for use in trademark processing.
|
||||
* `CommitLog*` -- These entities store the commit log information.
|
||||
* `Contact` -- These hold the ICANN contact information (but not
|
||||
registrar contacts, who have a separate entity type).
|
||||
* `Cursor` -- We use Cursor entities to maintain state about daily processes,
|
||||
remembering which dates have been processed. For instance, for the RDE
|
||||
export, Cursor entities maintain the date up to which each TLD has been
|
||||
exported.
|
||||
* `Domain` -- These hold the ICANN domain information.
|
||||
* `DomainRecord` -- These are used during the DNS update process.
|
||||
* `EntityGroupRoot` -- There is only one EntityGroupRoot entity, which serves
|
||||
as the Datastore parent of many other entities.
|
||||
* `EppResourceIndex` -- These entities allow enumeration of EPP resources
|
||||
(such as domains, hosts and contacts), which would otherwise be difficult to
|
||||
do in Datastore.
|
||||
* `ExceptionReportEntity` -- These entities are generated automatically by
|
||||
ECatcher, a Google-internal logging and debugging tool. Non-Google users
|
||||
should not encounter these entries.
|
||||
* `ForeignKeyContactIndex`, `ForeignKeyDomainIndex`, and
|
||||
`ForeignKeyHostIndex` -- These act as a unique index on contacts, domains
|
||||
and hosts, allowing transactional lookup by foreign key.
|
||||
* `HistoryEntry` -- A HistoryEntry is the record of a command which mutated an
|
||||
EPP resource. It serves as the parent of BillingEvents and PollMessages.
|
||||
* `HostRecord` -- These are used during the DNS update process.
|
||||
* `Host` -- These hold the ICANN host information.
|
||||
* `Lock` -- Lock entities are used to control access to a shared resource such
|
||||
as an App Engine queue. Under ordinary circumstances, these locks will be
|
||||
cleaned up automatically, and should not accumulate.
|
||||
* `MR-*` -- These entities are generated by the App Engine MapReduce library
|
||||
in the course of running MapReduces.
|
||||
* `Modification` -- A Modification is a special type of billing event which
|
||||
represents the modification of a OneTime billing event.
|
||||
* `OneTime` -- A OneTime is a billing event which represents a one-time charge
|
||||
or credit to the client (as opposed to Recurring).
|
||||
* `pipeline-*` -- These entities are also generated by the App Engine
|
||||
MapReduce library.
|
||||
* `PollMessage` -- PollMessages are generated by the system to notify
|
||||
registrars of asynchronous responses and status changes.
|
||||
* `PremiumList`, `PremiumListEntry`, and `PremiumListRevision` -- The standard
|
||||
method for determining which domain names receive premium pricing is to
|
||||
maintain a static list of premium names. Each PremiumList contains some
|
||||
number of PremiumListRevisions, each of which in turn contains a
|
||||
PremiumListEntry for each premium name.
|
||||
* `RdeRevision` -- These entities are used by the RDE subsystem in the process
|
||||
of generating files.
|
||||
* `Recurring` -- A Recurring is a billing event which represents a recurring
|
||||
charge to the client (as opposed to OneTime).
|
||||
* `Registrar` -- These hold information about client registrars.
|
||||
* `RegistrarPoc` -- Registrars have contacts just as domains do. These are
|
||||
stored in a special RegistrarPoc entity.
|
||||
* `Registry` -- These hold information about the TLDs supported by the
|
||||
Registry system.
|
||||
* `RegistryCursor` -- These entities are the predecessor to the Cursor
|
||||
entities. We are no longer using them, and will be deleting them soon.
|
||||
* `ReservedList` -- Each ReservedList entity represents an entire list of
|
||||
reserved names which cannot be registered. Each TLD can have one or more
|
||||
attached reserved lists.
|
||||
* `ServerSecret` -- this is a single entity containing the secret numbers used
|
||||
for generating tokens such as XSRF tokens.
|
||||
* `SignedMarkRevocationList` -- The entities together contain the Signed Mark
|
||||
Data Revocation List file downloaded from the TMCH MarksDB each day. Each
|
||||
entity contains up to 10,000 rows of the file, so depending on the size of
|
||||
the file, there will be some handful of entities.
|
||||
* `TmchCrl` -- This is a single entity containing ICANN's TMCH CA Certificate
|
||||
Revocation List.
|
||||
To be filled.
|
||||
|
||||
## Cloud Storage buckets
|
||||
|
||||
Nomulus uses [Cloud Storage](https://cloud.google.com/storage/) for bulk storage
|
||||
of large flat files that aren't suitable for Datastore. These files include
|
||||
backups, RDE exports, Datastore snapshots (for ingestion into BigQuery), and
|
||||
reports. Each bucket name must be unique across all of Google Cloud Storage, so
|
||||
we use the common recommended pattern of prefixing all buckets with the name of
|
||||
the App Engine app (which is itself globally unique). Most of the bucket names
|
||||
are configurable, but the defaults are as follows, with PROJECT standing in as a
|
||||
placeholder for the App Engine app name:
|
||||
of large flat files that aren't suitable for Cloud SQL. These files include
|
||||
backups, RDE exports, and reports. Each bucket name must be unique across all of
|
||||
Google Cloud Storage, so we use the common recommended pattern of prefixing all
|
||||
buckets with the name of the App Engine app (which is itself globally unique).
|
||||
Most of the bucket names are configurable, but the defaults are as follows, with
|
||||
PROJECT standing in as a placeholder for the App Engine app name:
|
||||
|
||||
* `PROJECT-billing` -- Monthly invoice files for each registrar.
|
||||
* `PROJECT-commits` -- Daily exports of commit logs that are needed for
|
||||
|
@ -421,9 +293,6 @@ placeholder for the App Engine app name:
|
|||
regularly uploaded to the escrow provider. Lifecycle is set to 90 days. The
|
||||
bucket must exist.
|
||||
* `PROJECT-reporting` -- Contains monthly ICANN reporting files.
|
||||
* `PROJECT-snapshots` -- Contains daily exports of Datastore entities of types
|
||||
defined in `ExportConstants.java`. These are imported into BigQuery daily to
|
||||
allow for in-depth querying.
|
||||
* `PROJECT.appspot.com` -- Temporary MapReduce files are stored here. By
|
||||
default, the App Engine MapReduce library places its temporary files in a
|
||||
bucket named {project}.appspot.com. This bucket must exist. To keep
|
||||
|
|
|
@ -77,14 +77,14 @@ The following cursor types are defined:
|
|||
* **`SYNC_REGISTRAR_SHEET`** - Tracks the last time the registrar spreadsheet
|
||||
was successfully synced.
|
||||
|
||||
All `Cursor` entities in Datastore contain a `DateTime` that represents the next
|
||||
timestamp at which an operation should resume processing and a `CursorType` that
|
||||
identifies which operation the cursor is associated with. In many cases, there
|
||||
are multiple cursors per operation; for instance, the cursors related to RDE
|
||||
reporting, staging, and upload are per-TLD cursors. To accomplish this, each
|
||||
All `Cursor` entities in the database contain a `DateTime` that represents the
|
||||
next timestamp at which an operation should resume processing and a `CursorType`
|
||||
that identifies which operation the cursor is associated with. In many cases,
|
||||
there are multiple cursors per operation; for instance, the cursors related to
|
||||
RDE reporting, staging, and upload are per-TLD cursors. To accomplish this, each
|
||||
`Cursor` also has a scope, a `Key<ImmutableObject>` to which the particular
|
||||
cursor applies (this can be e.g. a `Registry` or any other `ImmutableObject` in
|
||||
Datastore, depending on the operation). If the `Cursor` applies to the entire
|
||||
the database, depending on the operation). If the `Cursor` applies to the entire
|
||||
registry environment, it is considered a global cursor and has a scope of
|
||||
`EntityGroupRoot.getCrossTldKey()`.
|
||||
|
||||
|
@ -172,27 +172,14 @@ objects by their unique IDs:
|
|||
* `Host`: fully-qualified host name
|
||||
|
||||
Since all `EppResource` entities are indexed on ROID (which is also unique, but
|
||||
not as useful as the resource's name), a `ForeignKeyIndex` provides a way to
|
||||
not as useful as the resource's name), the `ForeignKeyUtils` provides a way to
|
||||
look up the resources using another key which is also unique during the lifetime
|
||||
of the resource (though not for all time).
|
||||
|
||||
A `ForeignKeyIndex` is updated as a resource is created or deleted. It is
|
||||
important to note that throughout the lifecycle of an `EppResource`, the
|
||||
underlying Datastore entity is never hard-deleted; its deletion time is set to
|
||||
the time at which the EPP command to delete the resource was set, and it remains
|
||||
in Datastore. Other resources with that same name can then be created.
|
||||
|
||||
## EPP resource index
|
||||
|
||||
An `EppResourceIndex` is an index that allows for quick enumeration of all
|
||||
`EppResource` entities in Datastore. Datastore does not otherwise provide an
|
||||
easy way to efficiently and strongly consistently enumerate all entities of a
|
||||
given type. Each `EppResourceIndex` is assigned randomly to an
|
||||
`EppResourceIndexBucket` upon creation, the number of which is configured to be
|
||||
greater than the number of shards typically used for Mapreduces that enumerate
|
||||
these entities. Mapreduces that process all `EppResource` entities (or
|
||||
subclasses thereof) distribute each `EppResourceIndexBucket` to available
|
||||
shards.
|
||||
It is important to note that throughout the lifecycle of an `EppResource`, the
|
||||
underlying entity is never hard-deleted; its deletion time is set to the time at
|
||||
which the EPP command to delete the resource was set, and it remains in the
|
||||
database. Other resources with that same name can then be created.
|
||||
|
||||
## History entries
|
||||
|
||||
|
@ -232,7 +219,7 @@ between the registry and registrars. Refer to [RFC 5730 Section
|
|||
2.9.2.3](https://tools.ietf.org/html/rfc5730#section-2.9.2.3) for their protocol
|
||||
specification.
|
||||
|
||||
Poll messages are stored by the system as entities in Datastore. All poll
|
||||
Poll messages are stored by the system as entities in the database. All poll
|
||||
messages have an event time at which they become active; any poll request before
|
||||
that time will not return the poll message. For example, every domain when
|
||||
created enqueues a speculative poll message for the automatic renewal of the
|
||||
|
|
|
@ -3,50 +3,6 @@
|
|||
This file is a motley assortment of informational emails generated in response
|
||||
to questions from development partners.
|
||||
|
||||
## Which entities are backed up using commit logs?
|
||||
|
||||
Short answer: Check the file `backup_kinds.txt`.
|
||||
|
||||
Long answer: There are really two axes. Our custom Ofy actually doesn't
|
||||
condition commit logging on the `@Entity` class being mutated. Rather, it
|
||||
conditions based on:
|
||||
|
||||
1. the methods you're invoking
|
||||
1. annotations on the class being mutated
|
||||
|
||||
**Methods:** The standard `save()` and `delete()` methods always write commit
|
||||
logs. The alternate `saveWithoutBackup()` and `deleteWithoutBackup()` methods
|
||||
never write commit logs (and have appropriately scary names to avoid accidental
|
||||
use). This makes it clear what you're getting at the callsite.
|
||||
|
||||
**Annotations:** There are regular `@Entity`-annotated entities,
|
||||
`@VirtualEntity`-annotated entities, and `@NotBackedUp`-annotated entities. An
|
||||
`@VirtualEntity` is a "virtual entity" that just serves to construct parent keys
|
||||
for other entities (e.g. `EppResourceIndexBucket`) and is never written to
|
||||
Datastore itself. An `@NotBackedUp`-annotated entity is one that specifically
|
||||
shouldn't be backed up (like the commit log entities themselves).
|
||||
|
||||
We don't actually prevent you from not-backing-up a regular entity, because
|
||||
sometimes that's necessary (e.g. in the restore logic, and some other
|
||||
miscellaneous places). We do prevent you from trying to save/delete an
|
||||
`@VirtualEntity` (with or without backups) or save/delete an `@NotBackedUp`
|
||||
entity with backups (i.e. using the regular `save()`/`delete()` calls), since
|
||||
these are always usage errors.
|
||||
|
||||
We went with an `@NotBackedUp` annotation versus an `@BackedUp` annotation
|
||||
largely because the not-backed-up case is more of the special case (as explained
|
||||
above, the annotation itself doesn't directly turn backups on or off). To keep
|
||||
track of what we're intending to back up, we:
|
||||
|
||||
1. synthesize a list of all the backed-up kinds by filtering the registered
|
||||
entity classes to remove those with `@VirtualEntity` and `@NotBackedUp`
|
||||
annotations:
|
||||
[`ExportConstants`](https://github.com/google/nomulus/blob/master/java/google/registry/export/ExportConstants.java#L82),
|
||||
1. check this list into the repo:
|
||||
[`backup_kinds.txt`](https://github.com/google/nomulus/blob/master/javatests/google/registry/export/backup_kinds.txt), then
|
||||
1. run an enforcement test to make sure it is up to date:
|
||||
[`ExportConstantsTest`](https://github.com/google/nomulus/blob/master/javatests/google/registry/export/ExportConstantsTest.java#L55).
|
||||
|
||||
## How do I mock Google Cloud Storage in tests?
|
||||
|
||||
AppEngine's GCS client automatically switches over to a local implementation,
|
||||
|
@ -142,31 +98,4 @@ specify .domain.tld after the nameserver wildcard (e.g. ns*.domain.tld). But you
|
|||
can't do anything else, like searching for nameservers with ns*.tld. When using
|
||||
a wildcard, we currently require a prefix of at least two characters, to avoid
|
||||
having someone search for *. There are other limitations to the system which we
|
||||
plan to address in the future.
|
||||
|
||||
## How do I embed a new class in an existing Datastore entity class?
|
||||
|
||||
In Objectify 4.x, which we are pinned to, you have to annotate a class with
|
||||
@Embed in order to natively nest it within an entity, e.g. as a field within an
|
||||
@Entity class. This behavior is different from how things work in Objectify 5.x,
|
||||
which is what the current documentation reflects. Unfortunately the 4.x docs are
|
||||
hard to access, but you can search this old copy for "embed" to see the details:
|
||||
|
||||
[https://raw.githubusercontent.com/objectify/objectify-legacy-wiki/v4/Entities.wiki](https://raw.githubusercontent.com/objectify/objectify-legacy-wiki/v4/Entities.wiki)
|
||||
|
||||
So you'd have to annotate your subobject with @Embed for it to work. It's
|
||||
possible in Objectify 4.x to have a class that's both @Embed and @Entity, but
|
||||
it'd be a little unorthodox. Usually it'd be cleaner to either make it purely
|
||||
@Embed (so purely a container for data within the parent entity) or purely
|
||||
@Entity, in which case you wouldn't use it as a field of the other @Entity but
|
||||
would instead refer to it by key, e.g. you'd store a Key<MySubObject> instead.
|
||||
For objects which don't change very often, it is preferable to use the @Entity
|
||||
approach.
|
||||
|
||||
## How do I resolve Bazel package visibility problems?
|
||||
|
||||
As a short-term solution, you can disable the bazel visibility checking with
|
||||
[`--nocheck_visibility`](https://www.bazel.io/versions/master/docs/bazel-user-manual.html#flag--check_visibility).
|
||||
We are investigating ways to clarify the way that custom functionality is
|
||||
expected to be added, which will involve changing the package visibility
|
||||
accordingly.
|
||||
plan to address in the future.
|
|
@ -4,7 +4,7 @@ There are multiple different kinds of configuration that go into getting a
|
|||
working registry system up and running. Broadly speaking, configuration works in
|
||||
two ways -- globally, for the entire sytem, and per-TLD. Global configuration is
|
||||
managed by editing code and deploying a new version, whereas per-TLD
|
||||
configuration is data that lives in Datastore in `Registry` entities, and is
|
||||
configuration is data that lives in the database in `Registry` entities, and is
|
||||
updated by running `nomulus` commands without having to deploy a new version.
|
||||
|
||||
## Initial configuration
|
||||
|
@ -39,7 +39,6 @@ The main files of note that come pre-configured in Nomulus are:
|
|||
* `web.xml` -- Configuration of URL paths on the webserver
|
||||
* `appengine-web.xml` -- Overall App Engine settings including number and type
|
||||
of instances
|
||||
* `datastore-indexes.xml` -- Configuration of entity indexes in Datastore
|
||||
* `queue.xml` -- Configuration of App Engine task queues
|
||||
* `application.xml` -- Configuration of the application name and its services
|
||||
|
||||
|
@ -49,8 +48,8 @@ doc, and the rest are covered in the general App Engine documentation.
|
|||
If you are not writing new code to implement custom features, is unlikely that
|
||||
you will need to make any modifications beyond simple changes to
|
||||
`application.xml` and `appengine-web.xml`. If you are writing new features, it's
|
||||
likely you'll need to add cronjobs, URL paths, Datastore indexes, and task
|
||||
queues, and thus edit those associated XML files.
|
||||
likely you'll need to add cronjobs, URL paths, and task queues, and thus edit
|
||||
those associated XML files.
|
||||
|
||||
The existing codebase is configured for running a full-scale registry with
|
||||
multiple TLDs. In order to deploy to App Engine, you will either need to
|
||||
|
@ -177,7 +176,7 @@ SecretManager to configure accordingly, for example:
|
|||
|
||||
## Per-TLD configuration
|
||||
|
||||
`Registry` entities, which are persisted to Datastore, are used for per-TLD
|
||||
`Registry` entities, which are persisted to the database, are used for per-TLD
|
||||
configuration. They contain any kind of configuration that is specific to a TLD,
|
||||
such as the create/renew price of a domain name, the pricing engine
|
||||
implementation, the DNS writer implementation, whether escrow exports are
|
||||
|
|
|
@ -34,61 +34,4 @@ will continue running until you terminate the process.
|
|||
If you are adding new URL paths, or new directories of web-accessible resources,
|
||||
you will need to make the corresponding changes in `RegistryTestServer`. This
|
||||
class contains all of the routing and static file information used by the local
|
||||
development server.
|
||||
|
||||
## Performing Datastore schema migrations
|
||||
|
||||
At some point in the development of a large product it will most likely become
|
||||
necessary to perform a schema migration on data persisted to the database.
|
||||
Because Datastore is a schema-less database, adding new fields is as simple as
|
||||
writing to them, but changing the names or data formats of existing fields
|
||||
requires more care, especially on a running production environment. The
|
||||
Objectify documentation has a good [introductory
|
||||
guide](https://github.com/objectify/objectify/wiki/SchemaMigration) to schema
|
||||
migrations that is worth reading. Beyond that, you may want to use some of the
|
||||
following techniques.
|
||||
|
||||
The requirements for a good schema migration are as follows:
|
||||
|
||||
* There must be no down-time or interruption of service.
|
||||
* The upgrade must be rollback-safe, i.e. the new version of the app can be
|
||||
reverted and everything will continue working.
|
||||
|
||||
In order to meet these requirements, a multiple-phase roll-out strategy is used
|
||||
as follows:
|
||||
|
||||
1. Dual-write, reading from old field. Add the new Datastore fields along with
|
||||
any required Datastore indexes. Use an `@OnSave` method on the entity to
|
||||
copy over the contents of the old field to the new field every time the
|
||||
entity is saved.
|
||||
2. Deploy the new version of the app.
|
||||
3. Re-save all affected entities in Datastore. For `EppResources` this can be
|
||||
accomplished by running
|
||||
[`ResaveAllEppResourcesAction`](https://github.com/google/nomulus/blob/master/java/google/registry/tools/server/ResaveAllEppResourcesAction.java);
|
||||
for other entities you may need to write something custom. Re-saving all
|
||||
entities forces the `@OnSave` method to fire for every entity, copying over
|
||||
the contents of the old fields to the new fields. Any additional entities
|
||||
that are created after the mapreduce is run will have the right values for
|
||||
the new field because the `@OnSave` method is writing them.
|
||||
4. Dual-write, now reading from new field. Switch over all places in the code
|
||||
that are using the data to read from the new fields rather than from the old
|
||||
fields. Adjust the `@OnSave` method so that it is copying over the contents
|
||||
from the new field to the old field. Dual-writing to the old fields ensures
|
||||
that it is safe to roll back to the prior version if necessary, since the
|
||||
data it is expecting will still be there.
|
||||
5. Deploy the new version of the app.
|
||||
6. Delete the old fields, their indexes, and the `@OnSave` method.
|
||||
7. Deploy the new version of the app. The schema migration is now complete.
|
||||
|
||||
The migration away from using a wrapper class around Keys on `Domain`
|
||||
objects is instructive as an example:
|
||||
|
||||
* [Step
|
||||
1](https://github.com/google/nomulus/commit/861fd60d2cb408ba2f9570d3881d316475b728c2),
|
||||
which implements dual-writing.
|
||||
* [Step
|
||||
4](https://github.com/google/nomulus/commit/361a53a3c985c14539e5ec1a31cf4ad192f67a5d),
|
||||
which switches over to using the new fields.
|
||||
* [Step
|
||||
6](https://github.com/google/nomulus/commit/780a5add78e735589f25f736059c29e9faf9aef5),
|
||||
which removes the old fields.
|
||||
development server.
|
|
@ -582,7 +582,7 @@ that window, this flow allows the losing client to explicitly approve the
|
|||
transfer request, which then becomes effective immediately.
|
||||
|
||||
When the transfer was requested, poll messages and billing events were saved to
|
||||
Datastore with timestamps such that they only would become active when the
|
||||
the database with timestamps such that they only would become active when the
|
||||
transfer period passed. In this flow, those speculative objects are deleted and
|
||||
replaced with new ones with the correct approval time.
|
||||
|
||||
|
@ -621,7 +621,7 @@ that window, this flow allows the gaining client to withdraw the transfer
|
|||
request.
|
||||
|
||||
When the transfer was requested, poll messages and billing events were saved to
|
||||
Datastore with timestamps such that they only would become active when the
|
||||
the database with timestamps such that they only would become active when the
|
||||
transfer period passed. In this flow, those speculative objects are deleted.
|
||||
|
||||
### Errors
|
||||
|
@ -705,8 +705,8 @@ default five days) after which the transfer is automatically approved. Within
|
|||
that window, the transfer might be approved explicitly by the losing registrar
|
||||
or rejected, and the gaining registrar can also cancel the transfer request.
|
||||
|
||||
When a transfer is requested, poll messages and billing events are saved to
|
||||
Datastore with timestamps such that they only become active when the
|
||||
When a transfer is requested, poll messages and billing events are saved to the
|
||||
database with timestamps such that they only become active when the
|
||||
server-approval period passes. Keys to these speculative objects are saved in
|
||||
the domain's transfer data, and on explicit approval, rejection or cancellation
|
||||
of the request, they will be deleted (and in the approval case, replaced with
|
||||
|
@ -1033,8 +1033,9 @@ An EPP flow for acknowledging {@link PollMessage}s.
|
|||
|
||||
Registrars refer to poll messages using an externally visible id generated by
|
||||
{@link PollMessageExternalKeyConverter}. One-time poll messages are deleted from
|
||||
Datastore once they are ACKed, whereas autorenew poll messages are simply marked
|
||||
as read, and won't be delivered again until the next year of their recurrence.
|
||||
the database once they are ACKed, whereas autorenew poll messages are simply
|
||||
marked as read, and won't be delivered again until the next year of their
|
||||
recurrence.
|
||||
|
||||
### Errors
|
||||
|
||||
|
|
|
@ -21,13 +21,13 @@ com.github.ben-manes.caffeine:caffeine:2.9.3=default,deploy_jar,runtimeClasspath
|
|||
com.github.docker-java:docker-java-api:3.2.13=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
com.github.docker-java:docker-java-transport-zerodep:3.2.13=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
com.github.docker-java:docker-java-transport:3.2.13=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jffi:1.3.9=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jffi:1.3.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-a64asm:1.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-constants:0.10.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-enxio:0.32.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-ffi:2.2.11=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-posix:3.1.15=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-unixsocket:0.38.17=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-constants:0.10.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-enxio:0.32.14=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-ffi:2.2.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-posix:3.1.16=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-unixsocket:0.38.19=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.jnr:jnr-x86asm:1.0.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
com.google.android:annotations:4.1.1.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -36,7 +36,7 @@ com.google.api-client:google-api-client-jackson2:1.32.2=default,deploy_jar,runti
|
|||
com.google.api-client:google-api-client-java6:1.35.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api-client:google-api-client-servlet:1.35.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api-client:google-api-client:1.35.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:gapic-google-cloud-storage-v2:2.16.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:gapic-google-cloud-storage-v2:2.17.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:2.23.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -47,9 +47,8 @@ com.google.api.grpc:grpc-google-cloud-pubsublite-v1:1.7.1=default,deploy_jar,run
|
|||
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-cloud-spanner-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-cloud-storage-v2:2.16.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-cloud-storage-v2:2.17.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-common-protos:2.9.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:grpc-google-iam-v1:1.6.22=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:2.23.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -60,23 +59,23 @@ com.google.api.grpc:proto-google-cloud-firestore-v1:3.6.0=default,deploy_jar,run
|
|||
com.google.api.grpc:proto-google-cloud-monitoring-v3:3.4.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-pubsub-v1:1.102.20=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-pubsublite-v1:1.7.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-secretmanager-v1:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-secretmanager-v1:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-spanner-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-storage-v2:2.16.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-tasks-v2:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.96.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.96.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-common-protos:2.11.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-iam-v1:1.6.22=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:api-common:2.2.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:gax-grpc:2.20.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:gax-httpjson:0.105.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:gax:2.20.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-storage-v2:2.17.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-tasks-v2:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.98.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.98.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-common-protos:2.12.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api.grpc:proto-google-iam-v1:1.7.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:api-common:2.4.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:gax-grpc:2.21.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:gax-httpjson:0.106.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.api:gax:2.21.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-appengine:v1-rev20221205-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-appengine:v1-rev20230109-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-bigquery:v2-rev20220924-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-clouddebugger:v2-rev20220318-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-cloudresourcemanager:v1-rev20220828-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -88,49 +87,49 @@ com.google.apis:google-api-services-healthcare:v1-rev20220818-2.0.0=default,depl
|
|||
com.google.apis:google-api-services-iamcredentials:v1-rev20210326-1.32.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-monitoring:v3-rev20221205-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-pubsub:v1-rev20220904-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-sheets:v4-rev20220927-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-sqladmin:v1beta4-rev20221017-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-sheets:v4-rev20221216-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-sqladmin:v1beta4-rev20221209-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.apis:google-api-services-storage:v1-rev20220705-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.appengine:appengine-api-1.0-sdk:1.9.86=compileClasspath,default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
com.google.appengine:appengine-testing:1.9.86=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auth:google-auth-library-credentials:1.13.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auth:google-auth-library-oauth2-http:1.13.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auth:google-auth-library-credentials:1.14.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auth:google-auth-library-oauth2-http:1.14.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auto.service:auto-service-annotations:1.0.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auto.value:auto-value-annotations:1.10.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auto.value:auto-value-annotations:1.8.1=testCompileClasspath
|
||||
com.google.auto.value:auto-value:1.9=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auto.value:auto-value:1.10.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.auto:auto-common:0.10=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
com.google.cloud.bigdataoss:gcsio:2.2.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.bigdataoss:util:2.2.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.bigtable:bigtable-client-core:1.26.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.bigtable:bigtable-metrics-api:1.26.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.datastore:datastore-v1-proto-client:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.sql:jdbc-socket-factory-core:1.7.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.sql:postgres-socket-factory:1.7.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.sql:jdbc-socket-factory-core:1.8.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud.sql:postgres-socket-factory:1.8.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-bigquerystorage:2.23.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-bigtable-stats:2.14.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-bigtable:2.14.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-core-grpc:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-core-http:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-core:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-core-grpc:2.9.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-core-http:2.9.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-core:2.9.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-firestore:3.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-monitoring:3.4.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-pubsub:1.120.20=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-pubsublite:1.7.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-secretmanager:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-secretmanager:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-spanner:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-storage:2.16.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-tasks:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-storage:2.17.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:google-cloud-tasks:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:grpc-gcp:1.2.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.cloud:proto-google-cloud-firestore-bundle-v1:3.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.code.findbugs:jFormatString:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
com.google.code.findbugs:jsr305:3.0.2=annotationProcessor,checkstyle,compileClasspath,default,deploy_jar,errorprone,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
|
||||
com.google.code.gson:gson:2.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.code.gson:gson:2.10.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.common.html.types:types:1.0.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.dagger:dagger:2.44.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.errorprone:error_prone_annotation:2.3.4=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
com.google.errorprone:error_prone_annotations:2.11.0=compileClasspath,testCompileClasspath
|
||||
com.google.errorprone:error_prone_annotations:2.16=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.errorprone:error_prone_annotations:2.18.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.errorprone:error_prone_annotations:2.3.4=annotationProcessor,checkstyle,errorprone,testAnnotationProcessor
|
||||
com.google.errorprone:error_prone_check_api:2.3.4=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
com.google.errorprone:error_prone_core:2.3.4=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
|
@ -164,8 +163,8 @@ com.google.oauth-client:google-oauth-client-java6:1.34.1=default,deploy_jar,runt
|
|||
com.google.oauth-client:google-oauth-client-jetty:1.34.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.oauth-client:google-oauth-client-servlet:1.34.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.oauth-client:google-oauth-client:1.34.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.protobuf:protobuf-java-util:3.21.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.protobuf:protobuf-java:3.21.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.protobuf:protobuf-java-util:3.21.12=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.protobuf:protobuf-java:3.21.12=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.protobuf:protobuf-java:3.4.0=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
com.google.re2j:re2j:1.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
com.google.re2j:re2j:1.7=compileClasspath,testCompileClasspath
|
||||
|
@ -195,22 +194,22 @@ io.confluent:kafka-schema-registry-client:5.3.2=default,deploy_jar,runtimeClassp
|
|||
io.dropwizard.metrics:metrics-core:3.1.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.github.classgraph:classgraph:4.8.104=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.github.java-diff-utils:java-diff-utils:4.12=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-alts:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-api:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-auth:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-alts:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-api:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-auth:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-census:1.49.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-context:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-core:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-googleapis:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-grpclb:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-netty-shaded:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-context:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-core:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-googleapis:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-grpclb:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-netty-shaded:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-netty:1.49.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-protobuf-lite:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-protobuf:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-protobuf-lite:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-protobuf:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-rls:1.49.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-services:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-stub:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-xds:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-services:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-stub:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.grpc:grpc-xds:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.netty:netty-buffer:4.1.77.Final=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.netty:netty-codec-http2:4.1.77.Final=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
io.netty:netty-codec-http:4.1.77.Final=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -278,8 +277,8 @@ org.apache.beam:beam-vendor-guava-26_0-jre:0.1=default,deploy_jar,runtimeClasspa
|
|||
org.apache.commons:commons-compress:1.22=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
org.apache.commons:commons-csv:1.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.apache.commons:commons-lang3:3.12.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.apache.httpcomponents:httpclient:4.5.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.apache.httpcomponents:httpcore:4.4.15=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.apache.httpcomponents:httpclient:4.5.14=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.apache.httpcomponents:httpcore:4.4.16=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath
|
||||
org.bouncycastle:bcpg-jdk15on:1.67=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.bouncycastle:bcpkix-jdk15on:1.67=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -290,7 +289,7 @@ org.checkerframework:checker-qual:2.11.1=checkstyle
|
|||
org.checkerframework:checker-qual:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
org.checkerframework:checker-qual:3.12.0=compileClasspath
|
||||
org.checkerframework:checker-qual:3.13.0=testCompileClasspath
|
||||
org.checkerframework:checker-qual:3.28.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.checkerframework:checker-qual:3.29.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.checkerframework:dataflow:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
org.checkerframework:javacutil:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
|
||||
org.codehaus.jackson:jackson-core-asl:1.9.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -305,7 +304,7 @@ org.eclipse.jetty:jetty-server:9.4.49.v20220914=default,deploy_jar,runtimeClassp
|
|||
org.eclipse.jetty:jetty-servlet:9.4.49.v20220914=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.eclipse.jetty:jetty-util-ajax:9.4.49.v20220914=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.eclipse.jetty:jetty-util:9.4.49.v20220914=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.flywaydb:flyway-core:9.10.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.flywaydb:flyway-core:9.11.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.glassfish.jaxb:jaxb-runtime:2.3.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.glassfish.jaxb:txw2:2.3.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.gwtproject:gwt-user:2.10.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -326,15 +325,15 @@ org.jetbrains:annotations:17.0.0=default,deploy_jar,runtimeClasspath,testCompile
|
|||
org.joda:joda-money:1.0.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.json:json:20200518=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.jsoup:jsoup:1.15.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-api:5.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-engine:5.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-commons:1.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-engine:1.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-launcher:1.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-runner:1.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-suite-api:1.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-suite-commons:1.9.1=testRuntimeClasspath
|
||||
org.junit:junit-bom:5.9.1=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-api:5.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.jupiter:junit-jupiter-engine:5.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-commons:1.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-engine:1.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-launcher:1.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-runner:1.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-suite-api:1.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.junit.platform:junit-platform-suite-commons:1.9.2=testRuntimeClasspath
|
||||
org.junit:junit-bom:5.9.2=testCompileClasspath,testRuntimeClasspath
|
||||
org.jvnet.staxex:stax-ex:1.8=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.opentest4j:opentest4j:1.2.0=testCompileClasspath,testRuntimeClasspath
|
||||
org.ow2.asm:asm-analysis:8.0.1=jacocoAnt
|
||||
|
@ -355,8 +354,8 @@ org.postgresql:postgresql:42.5.1=default,deploy_jar,runtimeClasspath,testRuntime
|
|||
org.reflections:reflections:0.9.12=checkstyle
|
||||
org.rnorth.duct-tape:duct-tape:1.0.8=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
org.slf4j:slf4j-api:1.7.36=testCompileClasspath
|
||||
org.slf4j:slf4j-api:2.0.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.slf4j:slf4j-jdk14:2.0.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.slf4j:slf4j-api:2.0.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.slf4j:slf4j-jdk14:2.0.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.springframework:spring-core:5.3.18=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.springframework:spring-expression:5.3.18=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.springframework:spring-jcl:5.3.18=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
@ -365,7 +364,7 @@ org.testcontainers:jdbc:1.17.6=default,deploy_jar,runtimeClasspath,testRuntimeCl
|
|||
org.testcontainers:junit-jupiter:1.17.6=testCompileClasspath,testRuntimeClasspath
|
||||
org.testcontainers:postgresql:1.17.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.testcontainers:testcontainers:1.17.6=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
|
||||
org.threeten:threetenbp:1.6.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.threeten:threetenbp:1.6.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.tukaani:xz:1.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.w3c.css:sac:1.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
org.xerial.snappy:snappy-java:1.1.8.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
|
||||
|
|
|
@ -40,12 +40,12 @@ is easier to exclude the suite classes than individual test classes. This is the
|
|||
reason why all test tasks in the :core project contain the exclude pattern
|
||||
'"**/*TestCase.*", "**/*TestSuite.*"'
|
||||
|
||||
Many Nomulus tests are not hermetic: they modify global state (e.g., the shared
|
||||
local instance of Datastore) but do not clean up on completion. This becomes a
|
||||
problem with Gradle. In the beginning we forced Gradle to run every test class
|
||||
in a new process, and incurred heavy overheads. Since then, we have fixed some
|
||||
tests, and manged to divide all tests into three suites that do not have
|
||||
intra-suite conflicts. We will revisit the remaining tests soon.
|
||||
Many Nomulus tests are not hermetic: they modify global state, but do not clean
|
||||
up on completion. This becomes a problem with Gradle. In the beginning we forced
|
||||
Gradle to run every test class in a new process, and incurred heavy overheads.
|
||||
Since then, we have fixed some tests, and manged to divide all tests into three
|
||||
suites that do not have intra-suite conflicts. We will revisit the remaining
|
||||
tests soon.
|
||||
|
||||
Note that it is unclear if all conflicting tests have been identified. More may
|
||||
be exposed if test execution order changes, e.g., when new tests are added or
|
||||
|
|
|
@ -86,35 +86,14 @@ agreement][registry-agreement].
|
|||
Nomulus provides [code to generate and send these
|
||||
deposits](./operational-procedures/rde-deposits.md).
|
||||
|
||||
### Monthly registry activity reporting
|
||||
### Monthly registry activity and transaction reporting
|
||||
|
||||
ICANN requires monthly activity reporting. The details are contained in
|
||||
Specification 3 of the [registry agreement][registry-agreement].
|
||||
ICANN requires monthly activity and transaction reporting. The details are
|
||||
contained in Specification 3 of the [registry agreement][registry-agreement].
|
||||
|
||||
These reports are generated by querying tables of Datastore snapshots which have
|
||||
been exported to the BigQuery data warehouse. The Nomulus release includes the
|
||||
queries used by Google. The script is Google-specific at the moment, but should
|
||||
be easy to modify to support a different instance of Nomulus.
|
||||
|
||||
The SQL queries which assemble the report are generated by the Python script
|
||||
[icann_report_query_builder.py](https://github.com/google/nomulus/blob/master/python/google/registry/reporting/icann_report_query_builder.py).
|
||||
The script generates a monolithic SQL query which is then executed against the
|
||||
BigQuery tables to generate a result table containing activity report data for
|
||||
all the TLDs in the system. With a little work, the data can be extracted into
|
||||
CSVs in the proper format.
|
||||
|
||||
The test script
|
||||
[icann_report_query_builder_test.py](https://github.com/google/nomulus/blob/master/python/google/registry/reporting/icann_report_query_builder_test.py)
|
||||
shows how the script is run, and the test data file
|
||||
[golden_activity_query.sql](https://github.com/google/nomulus/blob/master/python/google/registry/reporting/testdata/golden_activity_query.sql)
|
||||
shows what the resulting SQL looks like.
|
||||
|
||||
### Monthly transaction reporting
|
||||
|
||||
ICANN requires monthly transaction reporting. The details are also contained in
|
||||
Specification 3 of the [registry agreement][registry-agreement].
|
||||
|
||||
Transaction reporting is not currently part of the Nomulus release.
|
||||
These reports are mostly generated by querying the Cloud SQL database. There is
|
||||
currently a Google proprietary class to query DNS related activities that is
|
||||
not included in the open source Nomulus release.
|
||||
|
||||
### Zone File Access (ZFA)
|
||||
|
||||
|
|
|
@ -4,12 +4,12 @@
|
|||
|
||||
Nomulus comes with a `StaticPremiumListPricingEngine` that determines premium
|
||||
prices of domain labels (i.e. the part of the domain name without the TLD) by
|
||||
checking for their presence on a list of prices in Datastore. `nomulus` is used
|
||||
to load and update these lists from flat text files. The format of this list is
|
||||
simple: It is a newline-delimited CSV text file with each line containing the
|
||||
label and its price (including currency specifier in ISO-4217 format). Any
|
||||
individual label may not appear more than once in the file. Here's an example of
|
||||
the formatting:
|
||||
checking for their presence on a list of prices in the database. `nomulus` is
|
||||
used to load and update these lists from flat text files. The format of this
|
||||
list is simple: It is a newline-delimited CSV text file with each line
|
||||
containing the label and its price (including currency specifier in ISO-4217
|
||||
format). Any individual label may not appear more than once in the file. Here's
|
||||
an example of the formatting:
|
||||
|
||||
```
|
||||
premium,USD 100
|
||||
|
@ -28,7 +28,7 @@ location that is accessible to all members of your team (ideally in a source
|
|||
control system for revision tracking). These files should be thought of as the
|
||||
canonical versions of your premium lists. Note that there is no provided way to
|
||||
reconstruct a premium list .txt file from the premium list that is loaded into
|
||||
Datastore (though in principle it would be easy to do by writing a tool to do
|
||||
the database (though in principle it would be easy to do by writing a tool to do
|
||||
so), so don't lose those .txt files.
|
||||
|
||||
The nomulus repository contains an
|
||||
|
@ -37,7 +37,7 @@ The nomulus repository contains an
|
|||
## Creating a premium list
|
||||
|
||||
Once the file containing the premium prices is ready, run the
|
||||
`create_premium_list` command to load it into Datastore as follows:
|
||||
`create_premium_list` command to load it into the database as follows:
|
||||
|
||||
```shell
|
||||
$ nomulus -e {ENVIRONMENT} create_premium_list -n exampletld -i exampletld.txt
|
||||
|
@ -106,8 +106,9 @@ premiumList=Key<?>(EntityGroupRoot("cross-tld")/PremiumList("exampletld"))
|
|||
|
||||
## Listing all available premium lists
|
||||
|
||||
The `list_premium_lists` command is used to list all premium lists in Datastore.
|
||||
It takes no arguments and displays a simple list of premium lists as follows:
|
||||
The `list_premium_lists` command is used to list all premium lists in the
|
||||
database. It takes no arguments and displays a simple list of premium lists as
|
||||
follows:
|
||||
|
||||
```shell
|
||||
$ nomulus -e {ENVIRONMENT} list_premium_lists
|
||||
|
|
|
@ -26,7 +26,7 @@ need to implement a working and secure key store from which RDE can pull the
|
|||
private key used to transmit the deposits via sFTP.
|
||||
|
||||
For each phase and TLD in the process, the system maintains a `Cursor` entity in
|
||||
Datastore, which contains a timestamp indicating that everything before the
|
||||
the database, which contains a timestamp indicating that everything before the
|
||||
timestamp is current (except for RDE_UPLOAD_SFTP, which works a little
|
||||
differently). Only if the current time is after the cursor time do the actions
|
||||
check to see if they have work to do. For RDE, there are separate cursor types
|
||||
|
|
|
@ -82,7 +82,7 @@ reserved lists (see subsequent sections). The two naming patterns are thus:
|
|||
## Creating a reserved list
|
||||
|
||||
Once the file containing the list of reserved terms is created, run the
|
||||
`create_reserved_list` command to load it into Datastore as follows. For the
|
||||
`create_reserved_list` command to load it into the database as follows. For the
|
||||
purposes of this example, we are creating a common reserved list named
|
||||
"common_bad-words".
|
||||
|
||||
|
@ -163,8 +163,8 @@ reservedLists=[Key<?>(EntityGroupRoot("cross-tld")/ReservedList("common_bad-word
|
|||
|
||||
## Listing all available reserved lists
|
||||
|
||||
The `list_reserved_lists` command is used to list all reserved lists in
|
||||
Datastore. It takes no arguments and displays a simple list of reserved lists in
|
||||
The `list_reserved_lists` command is used to list all reserved lists in the
|
||||
database. It takes no arguments and displays a simple list of reserved lists in
|
||||
newline-delimited format as follows:
|
||||
|
||||
```shell
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue