zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-08-02 07:52:11 +02:00
Code Issues Projects Releases Packages Wiki Activity

Remove datastore related code (#1906)

Browse source
This commit is contained in:
Lai Jiang 2023-01-19 14:44:11 -05:00 • committed by GitHub
parent 913edb23ee
commit e41fd7877e
152 changed files with 886 additions and 4460 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/admin-tool.md
View file

@ -99,5 +99,5 @@ that are used by a large number of other tools are:
* **`ListObjectsCommand`** -- Lists all objects of a specific type from the
server and outputs them.
* **`MutatingCommand`** -- Provides a facility to create or update entities in
Datastore, and uses a diff algorithm to display the changes that will be
the database, and uses a diff algorithm to display the changes that will be
made before committing them.

147
docs/architecture.md
View file

@ -120,17 +120,6 @@ specification for the cron task.
Here are the task queues in use by the system. All are push queues unless
explicitly marked as otherwise.
* `async-delete-pull` and `async-host-rename-pull` -- Pull queues for tasks to
asynchronously delete contacts/hosts and to asynchronously refresh DNS for
renamed hosts, respectively. Tasks are enqueued during EPP flows and then
handled in batches by the regularly running cron tasks
`DeleteContactsAndHostsAction` and `RefreshDnsOnHostRenameAction`.
* `bigquery-streaming-metrics` -- Queue for metrics that are asynchronously
streamed to BigQuery in the `Metrics` class. Tasks are enqueued during EPP
flows in `EppController`. This means that there is a lag of a few seconds to
a few minutes between when metrics are generated and when they are queryable
in BigQuery, but this is preferable to slowing all EPP flows down and
blocking them on BigQuery streaming.
* `brda` -- Queue for tasks to upload weekly Bulk Registration Data Access
(BRDA) files to a location where they are available to ICANN. The
`RdeStagingReducer` (part of the RDE MapReduce) creates these tasks at the
@ -141,33 +130,6 @@ explicitly marked as otherwise.
`DnsWriter` for the TLD.
* `dns-publish` -- Queue for batches of DNS updates to be pushed to DNS
writers.
* `export-bigquery-poll` -- Queue for tasks to query the success/failure of a
given BigQuery export job. Tasks are enqueued by `BigqueryPollJobAction`.
* `export-commits` -- Queue for tasks to export commit log checkpoints. Tasks
are enqueued by `CommitLogCheckpointAction` (which is run every minute by
cron) and executed by `ExportCommitLogDiffAction`.
* `export-snapshot` -- Cron and push queue for tasks to load a Datastore
snapshot that was stored in Google Cloud Storage and export it to BigQuery.
Tasks are enqueued by both cron and `CheckSnapshotAction` and are executed
by both `ExportSnapshotAction` and `LoadSnapshotAction`.
* `export-snapshot-poll` -- Queue for tasks to check that a Datastore snapshot
has been successfully uploaded to Google Cloud Storage (this is an
asynchronous background operation that can take an indeterminate amount of
time). Once the snapshot is successfully uploaded, it is imported into
BigQuery. Tasks are enqueued by `ExportSnapshotAction` and executed by
`CheckSnapshotAction`.
* `export-snapshot-update-view` -- Queue for tasks to update the BigQuery
views to point to the most recently uploaded snapshot. Tasks are enqueued by
`LoadSnapshotAction` and executed by `UpdateSnapshotViewAction`.
* `group-members-sync` -- Cron queue for tasks to sync registrar contacts (not
domain contacts!) to Google Groups. Tasks are executed by
`SyncGroupMembersAction`.
* `load[0-9]` -- Queues used to load-test the system by `LoadTestAction`.
These queues don't need to exist except when actively running load tests
(running load tests on production environments is not recommended). There
are ten of these queues to provide simple sharding, because Nomulus is
capable of handling significantly more Queries Per Second than the highest
throttle limit available on task queues (which is 500 qps).
* `lordn-claims` and `lordn-sunrise` -- Pull queues for handling LORDN
exports. Tasks are enqueued synchronously during EPP commands depending on
whether the domain name in question has a claims notice ID.
@ -298,109 +260,19 @@ of experience running a production registry using this codebase.
errors, it can be pushed to Production.
5. Repeat once weekly, or potentially more often.
## Cloud Datastore
## Cloud SQL
Nomulus uses [Cloud
Datastore](https://cloud.google.com/appengine/docs/java/datastore/) as its
primary database. Cloud Datastore is a NoSQL document database that provides
automatic horizontal scaling, high performance, and high availability. All
information that is persisted to Cloud Datastore takes the form of Java classes
annotated with `@Entity` that are located in the `model` package. The [Objectify
library](https://cloud.google.com/appengine/docs/java/gettingstarted/using-datastore-objectify)
is used to persist instances of these classes in a format that Datastore
understands.
A brief overview of the different entity types found in the App Engine Datastore
Viewer may help administrators understand what they are seeing. Note that some
of these entities are part of App Engine tools that are outside of the domain
registry codebase:
* `_AE_*` -- These entities are created by App Engine.
* `_ah_SESSION` -- These entities track App Engine client sessions.
* `_GAE_MR_*` -- These entities are generated by App Engine while running
MapReduces.
* `BackupStatus` -- There should only be one of these entities, used to
maintain the state of the backup process.
* `Cancellation` -- A cancellation is a special type of billing event which
represents the cancellation of another billing event such as a OneTime or
Recurring.
* `ClaimsList`, `ClaimsListShard`, and `ClaimsListSingleton` -- These entities
store the TMCH claims list, for use in trademark processing.
* `CommitLog*` -- These entities store the commit log information.
* `Contact` -- These hold the ICANN contact information (but not
registrar contacts, who have a separate entity type).
* `Cursor` -- We use Cursor entities to maintain state about daily processes,
remembering which dates have been processed. For instance, for the RDE
export, Cursor entities maintain the date up to which each TLD has been
exported.
* `Domain` -- These hold the ICANN domain information.
* `DomainRecord` -- These are used during the DNS update process.
* `EntityGroupRoot` -- There is only one EntityGroupRoot entity, which serves
as the Datastore parent of many other entities.
* `EppResourceIndex` -- These entities allow enumeration of EPP resources
(such as domains, hosts and contacts), which would otherwise be difficult to
do in Datastore.
* `ExceptionReportEntity` -- These entities are generated automatically by
ECatcher, a Google-internal logging and debugging tool. Non-Google users
should not encounter these entries.
* `ForeignKeyContactIndex`, `ForeignKeyDomainIndex`, and
`ForeignKeyHostIndex` -- These act as a unique index on contacts, domains
and hosts, allowing transactional lookup by foreign key.
* `HistoryEntry` -- A HistoryEntry is the record of a command which mutated an
EPP resource. It serves as the parent of BillingEvents and PollMessages.
* `HostRecord` -- These are used during the DNS update process.
* `Host` -- These hold the ICANN host information.
* `Lock` -- Lock entities are used to control access to a shared resource such
as an App Engine queue. Under ordinary circumstances, these locks will be
cleaned up automatically, and should not accumulate.
* `MR-*` -- These entities are generated by the App Engine MapReduce library
in the course of running MapReduces.
* `Modification` -- A Modification is a special type of billing event which
represents the modification of a OneTime billing event.
* `OneTime` -- A OneTime is a billing event which represents a one-time charge
or credit to the client (as opposed to Recurring).
* `pipeline-*` -- These entities are also generated by the App Engine
MapReduce library.
* `PollMessage` -- PollMessages are generated by the system to notify
registrars of asynchronous responses and status changes.
* `PremiumList`, `PremiumListEntry`, and `PremiumListRevision` -- The standard
method for determining which domain names receive premium pricing is to
maintain a static list of premium names. Each PremiumList contains some
number of PremiumListRevisions, each of which in turn contains a
PremiumListEntry for each premium name.
* `RdeRevision` -- These entities are used by the RDE subsystem in the process
of generating files.
* `Recurring` -- A Recurring is a billing event which represents a recurring
charge to the client (as opposed to OneTime).
* `Registrar` -- These hold information about client registrars.
* `RegistrarPoc` -- Registrars have contacts just as domains do. These are
stored in a special RegistrarPoc entity.
* `Registry` -- These hold information about the TLDs supported by the
Registry system.
* `RegistryCursor` -- These entities are the predecessor to the Cursor
entities. We are no longer using them, and will be deleting them soon.
* `ReservedList` -- Each ReservedList entity represents an entire list of
reserved names which cannot be registered. Each TLD can have one or more
attached reserved lists.
* `ServerSecret` -- this is a single entity containing the secret numbers used
for generating tokens such as XSRF tokens.
* `SignedMarkRevocationList` -- The entities together contain the Signed Mark
Data Revocation List file downloaded from the TMCH MarksDB each day. Each
entity contains up to 10,000 rows of the file, so depending on the size of
the file, there will be some handful of entities.
* `TmchCrl` -- This is a single entity containing ICANN's TMCH CA Certificate
Revocation List.
To be filled.
## Cloud Storage buckets
Nomulus uses [Cloud Storage](https://cloud.google.com/storage/) for bulk storage
of large flat files that aren't suitable for Datastore. These files include
backups, RDE exports, Datastore snapshots (for ingestion into BigQuery), and
reports. Each bucket name must be unique across all of Google Cloud Storage, so
we use the common recommended pattern of prefixing all buckets with the name of
the App Engine app (which is itself globally unique). Most of the bucket names
are configurable, but the defaults are as follows, with PROJECT standing in as a
placeholder for the App Engine app name:
of large flat files that aren't suitable for Cloud SQL. These files include
backups, RDE exports, and reports. Each bucket name must be unique across all of
Google Cloud Storage, so we use the common recommended pattern of prefixing all
buckets with the name of the App Engine app (which is itself globally unique).
Most of the bucket names are configurable, but the defaults are as follows, with
PROJECT standing in as a placeholder for the App Engine app name:
* `PROJECT-billing` -- Monthly invoice files for each registrar.
* `PROJECT-commits` -- Daily exports of commit logs that are needed for
@ -421,9 +293,6 @@ placeholder for the App Engine app name:
regularly uploaded to the escrow provider. Lifecycle is set to 90 days. The
bucket must exist.
* `PROJECT-reporting` -- Contains monthly ICANN reporting files.
* `PROJECT-snapshots` -- Contains daily exports of Datastore entities of types
defined in `ExportConstants.java`. These are imported into BigQuery daily to
allow for in-depth querying.
* `PROJECT.appspot.com` -- Temporary MapReduce files are stored here. By
default, the App Engine MapReduce library places its temporary files in a
bucket named {project}.appspot.com. This bucket must exist. To keep

37
docs/code-structure.md
View file

@ -77,14 +77,14 @@ The following cursor types are defined:
* **`SYNC_REGISTRAR_SHEET`** - Tracks the last time the registrar spreadsheet
was successfully synced.
All `Cursor` entities in Datastore contain a `DateTime` that represents the next
timestamp at which an operation should resume processing and a `CursorType` that
identifies which operation the cursor is associated with. In many cases, there
are multiple cursors per operation; for instance, the cursors related to RDE
reporting, staging, and upload are per-TLD cursors. To accomplish this, each
All `Cursor` entities in the database contain a `DateTime` that represents the
next timestamp at which an operation should resume processing and a `CursorType`
that identifies which operation the cursor is associated with. In many cases,
there are multiple cursors per operation; for instance, the cursors related to
RDE reporting, staging, and upload are per-TLD cursors. To accomplish this, each
`Cursor` also has a scope, a `Key<ImmutableObject>` to which the particular
cursor applies (this can be e.g. a `Registry` or any other `ImmutableObject` in
Datastore, depending on the operation). If the `Cursor` applies to the entire
the database, depending on the operation). If the `Cursor` applies to the entire
registry environment, it is considered a global cursor and has a scope of
`EntityGroupRoot.getCrossTldKey()`.
@ -172,27 +172,14 @@ objects by their unique IDs:
* `Host`: fully-qualified host name
Since all `EppResource` entities are indexed on ROID (which is also unique, but
not as useful as the resource's name), a `ForeignKeyIndex` provides a way to
not as useful as the resource's name), the `ForeignKeyUtils` provides a way to
look up the resources using another key which is also unique during the lifetime
of the resource (though not for all time).
A `ForeignKeyIndex` is updated as a resource is created or deleted. It is
important to note that throughout the lifecycle of an `EppResource`, the
underlying Datastore entity is never hard-deleted; its deletion time is set to
the time at which the EPP command to delete the resource was set, and it remains
in Datastore. Other resources with that same name can then be created.
## EPP resource index
An `EppResourceIndex` is an index that allows for quick enumeration of all
`EppResource` entities in Datastore. Datastore does not otherwise provide an
easy way to efficiently and strongly consistently enumerate all entities of a
given type. Each `EppResourceIndex` is assigned randomly to an
`EppResourceIndexBucket` upon creation, the number of which is configured to be
greater than the number of shards typically used for Mapreduces that enumerate
these entities. Mapreduces that process all `EppResource` entities (or
subclasses thereof) distribute each `EppResourceIndexBucket` to available
shards.
It is important to note that throughout the lifecycle of an `EppResource`, the
underlying entity is never hard-deleted; its deletion time is set to the time at
which the EPP command to delete the resource was set, and it remains in the
database. Other resources with that same name can then be created.
## History entries
@ -232,7 +219,7 @@ between the registry and registrars. Refer to [RFC 5730 Section
2.9.2.3](https://tools.ietf.org/html/rfc5730#section-2.9.2.3) for their protocol
specification.
Poll messages are stored by the system as entities in Datastore. All poll
Poll messages are stored by the system as entities in the database. All poll
messages have an event time at which they become active; any poll request before
that time will not return the poll message. For example, every domain when
created enqueues a speculative poll message for the automatic renewal of the

73
docs/coding-faq.md
View file

@ -3,50 +3,6 @@
This file is a motley assortment of informational emails generated in response
to questions from development partners.
## Which entities are backed up using commit logs?
Short answer: Check the file `backup_kinds.txt`.
Long answer: There are really two axes. Our custom Ofy actually doesn't
condition commit logging on the `@Entity` class being mutated. Rather, it
conditions based on:
1. the methods you're invoking
1. annotations on the class being mutated
**Methods:** The standard `save()` and `delete()` methods always write commit
logs. The alternate `saveWithoutBackup()` and `deleteWithoutBackup()` methods
never write commit logs (and have appropriately scary names to avoid accidental
use). This makes it clear what you're getting at the callsite.
**Annotations:** There are regular `@Entity`-annotated entities,
`@VirtualEntity`-annotated entities, and `@NotBackedUp`-annotated entities. An
`@VirtualEntity` is a "virtual entity" that just serves to construct parent keys
for other entities (e.g. `EppResourceIndexBucket`) and is never written to
Datastore itself. An `@NotBackedUp`-annotated entity is one that specifically
shouldn't be backed up (like the commit log entities themselves).
We don't actually prevent you from not-backing-up a regular entity, because
sometimes that's necessary (e.g. in the restore logic, and some other
miscellaneous places). We do prevent you from trying to save/delete an
`@VirtualEntity` (with or without backups) or save/delete an `@NotBackedUp`
entity with backups (i.e. using the regular `save()`/`delete()` calls), since
these are always usage errors.
We went with an `@NotBackedUp` annotation versus an `@BackedUp` annotation
largely because the not-backed-up case is more of the special case (as explained
above, the annotation itself doesn't directly turn backups on or off). To keep
track of what we're intending to back up, we:
1. synthesize a list of all the backed-up kinds by filtering the registered
entity classes to remove those with `@VirtualEntity` and `@NotBackedUp`
annotations:
[`ExportConstants`](https://github.com/google/nomulus/blob/master/java/google/registry/export/ExportConstants.java#L82),
1. check this list into the repo:
[`backup_kinds.txt`](https://github.com/google/nomulus/blob/master/javatests/google/registry/export/backup_kinds.txt), then
1. run an enforcement test to make sure it is up to date:
[`ExportConstantsTest`](https://github.com/google/nomulus/blob/master/javatests/google/registry/export/ExportConstantsTest.java#L55).
## How do I mock Google Cloud Storage in tests?
AppEngine's GCS client automatically switches over to a local implementation,
@ -142,31 +98,4 @@ specify .domain.tld after the nameserver wildcard (e.g. ns*.domain.tld). But you
can't do anything else, like searching for nameservers with ns*.tld. When using
a wildcard, we currently require a prefix of at least two characters, to avoid
having someone search for *. There are other limitations to the system which we
plan to address in the future.
## How do I embed a new class in an existing Datastore entity class?
In Objectify 4.x, which we are pinned to, you have to annotate a class with
@Embed in order to natively nest it within an entity, e.g. as a field within an
@Entity class. This behavior is different from how things work in Objectify 5.x,
which is what the current documentation reflects. Unfortunately the 4.x docs are
hard to access, but you can search this old copy for "embed" to see the details:
[https://raw.githubusercontent.com/objectify/objectify-legacy-wiki/v4/Entities.wiki](https://raw.githubusercontent.com/objectify/objectify-legacy-wiki/v4/Entities.wiki)
So you'd have to annotate your subobject with @Embed for it to work. It's
possible in Objectify 4.x to have a class that's both @Embed and @Entity, but
it'd be a little unorthodox. Usually it'd be cleaner to either make it purely
@Embed (so purely a container for data within the parent entity) or purely
@Entity, in which case you wouldn't use it as a field of the other @Entity but
would instead refer to it by key, e.g. you'd store a Key<MySubObject> instead.
For objects which don't change very often, it is preferable to use the @Entity
approach.
## How do I resolve Bazel package visibility problems?
As a short-term solution, you can disable the bazel visibility checking with
[`--nocheck_visibility`](https://www.bazel.io/versions/master/docs/bazel-user-manual.html#flag--check_visibility).
We are investigating ways to clarify the way that custom functionality is
expected to be added, which will involve changing the package visibility
accordingly.
plan to address in the future.

9
docs/configuration.md
View file

@ -4,7 +4,7 @@ There are multiple different kinds of configuration that go into getting a
working registry system up and running. Broadly speaking, configuration works in
two ways -- globally, for the entire sytem, and per-TLD. Global configuration is
managed by editing code and deploying a new version, whereas per-TLD
configuration is data that lives in Datastore in `Registry` entities, and is
configuration is data that lives in the database in `Registry` entities, and is
updated by running `nomulus` commands without having to deploy a new version.
## Initial configuration
@ -39,7 +39,6 @@ The main files of note that come pre-configured in Nomulus are:
* `web.xml` -- Configuration of URL paths on the webserver
* `appengine-web.xml` -- Overall App Engine settings including number and type
of instances
* `datastore-indexes.xml` -- Configuration of entity indexes in Datastore
* `queue.xml` -- Configuration of App Engine task queues
* `application.xml` -- Configuration of the application name and its services
@ -49,8 +48,8 @@ doc, and the rest are covered in the general App Engine documentation.
If you are not writing new code to implement custom features, is unlikely that
you will need to make any modifications beyond simple changes to
`application.xml` and `appengine-web.xml`. If you are writing new features, it's
likely you'll need to add cronjobs, URL paths, Datastore indexes, and task
queues, and thus edit those associated XML files.
likely you'll need to add cronjobs, URL paths, and task queues, and thus edit
those associated XML files.
The existing codebase is configured for running a full-scale registry with
multiple TLDs. In order to deploy to App Engine, you will either need to
@ -177,7 +176,7 @@ SecretManager to configure accordingly, for example:
## Per-TLD configuration
`Registry` entities, which are persisted to Datastore, are used for per-TLD
`Registry` entities, which are persisted to the database, are used for per-TLD
configuration. They contain any kind of configuration that is specific to a TLD,
such as the create/renew price of a domain name, the pricing engine
implementation, the DNS writer implementation, whether escrow exports are

59
docs/developing.md
View file

@ -34,61 +34,4 @@ will continue running until you terminate the process.
If you are adding new URL paths, or new directories of web-accessible resources,
you will need to make the corresponding changes in `RegistryTestServer`. This
class contains all of the routing and static file information used by the local
development server.
## Performing Datastore schema migrations
At some point in the development of a large product it will most likely become
necessary to perform a schema migration on data persisted to the database.
Because Datastore is a schema-less database, adding new fields is as simple as
writing to them, but changing the names or data formats of existing fields
requires more care, especially on a running production environment. The
Objectify documentation has a good [introductory
guide](https://github.com/objectify/objectify/wiki/SchemaMigration) to schema
migrations that is worth reading. Beyond that, you may want to use some of the
following techniques.
The requirements for a good schema migration are as follows:
* There must be no down-time or interruption of service.
* The upgrade must be rollback-safe, i.e. the new version of the app can be
reverted and everything will continue working.
In order to meet these requirements, a multiple-phase roll-out strategy is used
as follows:
1. Dual-write, reading from old field. Add the new Datastore fields along with
any required Datastore indexes. Use an `@OnSave` method on the entity to
copy over the contents of the old field to the new field every time the
entity is saved.
2. Deploy the new version of the app.
3. Re-save all affected entities in Datastore. For `EppResources` this can be
accomplished by running
[`ResaveAllEppResourcesAction`](https://github.com/google/nomulus/blob/master/java/google/registry/tools/server/ResaveAllEppResourcesAction.java);
for other entities you may need to write something custom. Re-saving all
entities forces the `@OnSave` method to fire for every entity, copying over
the contents of the old fields to the new fields. Any additional entities
that are created after the mapreduce is run will have the right values for
the new field because the `@OnSave` method is writing them.
4. Dual-write, now reading from new field. Switch over all places in the code
that are using the data to read from the new fields rather than from the old
fields. Adjust the `@OnSave` method so that it is copying over the contents
from the new field to the old field. Dual-writing to the old fields ensures
that it is safe to roll back to the prior version if necessary, since the
data it is expecting will still be there.
5. Deploy the new version of the app.
6. Delete the old fields, their indexes, and the `@OnSave` method.
7. Deploy the new version of the app. The schema migration is now complete.
The migration away from using a wrapper class around Keys on `Domain`
objects is instructive as an example:
* [Step
1](https://github.com/google/nomulus/commit/861fd60d2cb408ba2f9570d3881d316475b728c2),
which implements dual-writing.
* [Step
4](https://github.com/google/nomulus/commit/361a53a3c985c14539e5ec1a31cf4ad192f67a5d),
which switches over to using the new fields.
* [Step
6](https://github.com/google/nomulus/commit/780a5add78e735589f25f736059c29e9faf9aef5),
which removes the old fields.
development server.

13
docs/flows.md
View file

@ -582,7 +582,7 @@ that window, this flow allows the losing client to explicitly approve the
transfer request, which then becomes effective immediately.
When the transfer was requested, poll messages and billing events were saved to
Datastore with timestamps such that they only would become active when the
the database with timestamps such that they only would become active when the
transfer period passed. In this flow, those speculative objects are deleted and
replaced with new ones with the correct approval time.
@ -621,7 +621,7 @@ that window, this flow allows the gaining client to withdraw the transfer
request.
When the transfer was requested, poll messages and billing events were saved to
Datastore with timestamps such that they only would become active when the
the database with timestamps such that they only would become active when the
transfer period passed. In this flow, those speculative objects are deleted.
### Errors
@ -705,8 +705,8 @@ default five days) after which the transfer is automatically approved. Within
that window, the transfer might be approved explicitly by the losing registrar
or rejected, and the gaining registrar can also cancel the transfer request.
When a transfer is requested, poll messages and billing events are saved to
Datastore with timestamps such that they only become active when the
When a transfer is requested, poll messages and billing events are saved to the
database with timestamps such that they only become active when the
server-approval period passes. Keys to these speculative objects are saved in
the domain's transfer data, and on explicit approval, rejection or cancellation
of the request, they will be deleted (and in the approval case, replaced with
@ -1033,8 +1033,9 @@ An EPP flow for acknowledging {@link PollMessage}s.
Registrars refer to poll messages using an externally visible id generated by
{@link PollMessageExternalKeyConverter}. One-time poll messages are deleted from
Datastore once they are ACKed, whereas autorenew poll messages are simply marked
as read, and won't be delivered again until the next year of their recurrence.
the database once they are ACKed, whereas autorenew poll messages are simply
marked as read, and won't be delivered again until the next year of their
recurrence.
### Errors

135
docs/gradle.lockfile
View file

@ -21,13 +21,13 @@ com.github.ben-manes.caffeine:caffeine:2.9.3=default,deploy_jar,runtimeClasspath
com.github.docker-java:docker-java-api:3.2.13=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
com.github.docker-java:docker-java-transport-zerodep:3.2.13=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
com.github.docker-java:docker-java-transport:3.2.13=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
com.github.jnr:jffi:1.3.9=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jffi:1.3.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-a64asm:1.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-constants:0.10.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-enxio:0.32.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-ffi:2.2.11=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-posix:3.1.15=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-unixsocket:0.38.17=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-constants:0.10.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-enxio:0.32.14=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-ffi:2.2.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-posix:3.1.16=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-unixsocket:0.38.19=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.jnr:jnr-x86asm:1.0.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,errorprone,testAnnotationProcessor
com.google.android:annotations:4.1.1.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -36,7 +36,7 @@ com.google.api-client:google-api-client-jackson2:1.32.2=default,deploy_jar,runti
com.google.api-client:google-api-client-java6:1.35.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api-client:google-api-client-servlet:1.35.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api-client:google-api-client:1.35.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:gapic-google-cloud-storage-v2:2.16.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:gapic-google-cloud-storage-v2:2.17.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:2.23.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -47,9 +47,8 @@ com.google.api.grpc:grpc-google-cloud-pubsublite-v1:1.7.1=default,deploy_jar,run
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-cloud-spanner-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-cloud-storage-v2:2.16.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-cloud-storage-v2:2.17.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-common-protos:2.9.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:grpc-google-iam-v1:1.6.22=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:2.23.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:0.147.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -60,23 +59,23 @@ com.google.api.grpc:proto-google-cloud-firestore-v1:3.6.0=default,deploy_jar,run
com.google.api.grpc:proto-google-cloud-monitoring-v3:3.4.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-pubsub-v1:1.102.20=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-pubsublite-v1:1.7.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-secretmanager-v1:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-secretmanager-v1:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-spanner-v1:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-storage-v2:2.16.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-tasks-v2:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.96.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.96.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-common-protos:2.11.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-iam-v1:1.6.22=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:api-common:2.2.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:gax-grpc:2.20.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:gax-httpjson:0.105.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:gax:2.20.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-storage-v2:2.17.0-alpha=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-tasks-v2:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.98.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.98.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-common-protos:2.12.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api.grpc:proto-google-iam-v1:1.7.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:api-common:2.4.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:gax-grpc:2.21.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:gax-httpjson:0.106.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.api:gax:2.21.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-appengine:v1-rev20221205-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-appengine:v1-rev20230109-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-bigquery:v2-rev20220924-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-clouddebugger:v2-rev20220318-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-cloudresourcemanager:v1-rev20220828-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -88,49 +87,49 @@ com.google.apis:google-api-services-healthcare:v1-rev20220818-2.0.0=default,depl
com.google.apis:google-api-services-iamcredentials:v1-rev20210326-1.32.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-monitoring:v3-rev20221205-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-pubsub:v1-rev20220904-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-sheets:v4-rev20220927-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-sqladmin:v1beta4-rev20221017-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-sheets:v4-rev20221216-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-sqladmin:v1beta4-rev20221209-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.apis:google-api-services-storage:v1-rev20220705-2.0.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.appengine:appengine-api-1.0-sdk:1.9.86=compileClasspath,default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
com.google.appengine:appengine-testing:1.9.86=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auth:google-auth-library-credentials:1.13.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auth:google-auth-library-oauth2-http:1.13.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auth:google-auth-library-credentials:1.14.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auth:google-auth-library-oauth2-http:1.14.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auto.service:auto-service-annotations:1.0.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auto.value:auto-value-annotations:1.10.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auto.value:auto-value-annotations:1.8.1=testCompileClasspath
com.google.auto.value:auto-value:1.9=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auto.value:auto-value:1.10.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.auto:auto-common:0.10=annotationProcessor,errorprone,testAnnotationProcessor
com.google.cloud.bigdataoss:gcsio:2.2.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.bigdataoss:util:2.2.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.bigtable:bigtable-client-core:1.26.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.bigtable:bigtable-metrics-api:1.26.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.datastore:datastore-v1-proto-client:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.sql:jdbc-socket-factory-core:1.7.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.sql:postgres-socket-factory:1.7.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.sql:jdbc-socket-factory-core:1.8.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud.sql:postgres-socket-factory:1.8.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-bigquerystorage:2.23.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-bigtable-stats:2.14.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-bigtable:2.14.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-core-grpc:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-core-http:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-core:2.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-core-grpc:2.9.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-core-http:2.9.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-core:2.9.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-firestore:3.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-monitoring:3.4.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-pubsub:1.120.20=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-pubsublite:1.7.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-secretmanager:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-secretmanager:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-spanner:6.31.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-storage:2.16.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-tasks:2.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-storage:2.17.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:google-cloud-tasks:2.8.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:grpc-gcp:1.2.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.cloud:proto-google-cloud-firestore-bundle-v1:3.6.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.code.findbugs:jFormatString:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
com.google.code.findbugs:jsr305:3.0.2=annotationProcessor,checkstyle,compileClasspath,default,deploy_jar,errorprone,runtimeClasspath,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
com.google.code.gson:gson:2.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.code.gson:gson:2.10.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.common.html.types:types:1.0.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.dagger:dagger:2.44.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.errorprone:error_prone_annotation:2.3.4=annotationProcessor,errorprone,testAnnotationProcessor
com.google.errorprone:error_prone_annotations:2.11.0=compileClasspath,testCompileClasspath
com.google.errorprone:error_prone_annotations:2.16=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.errorprone:error_prone_annotations:2.18.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.errorprone:error_prone_annotations:2.3.4=annotationProcessor,checkstyle,errorprone,testAnnotationProcessor
com.google.errorprone:error_prone_check_api:2.3.4=annotationProcessor,errorprone,testAnnotationProcessor
com.google.errorprone:error_prone_core:2.3.4=annotationProcessor,errorprone,testAnnotationProcessor
@ -164,8 +163,8 @@ com.google.oauth-client:google-oauth-client-java6:1.34.1=default,deploy_jar,runt
com.google.oauth-client:google-oauth-client-jetty:1.34.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.oauth-client:google-oauth-client-servlet:1.34.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.oauth-client:google-oauth-client:1.34.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.protobuf:protobuf-java-util:3.21.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.protobuf:protobuf-java:3.21.10=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.protobuf:protobuf-java-util:3.21.12=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.protobuf:protobuf-java:3.21.12=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.protobuf:protobuf-java:3.4.0=annotationProcessor,errorprone,testAnnotationProcessor
com.google.re2j:re2j:1.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
com.google.re2j:re2j:1.7=compileClasspath,testCompileClasspath
@ -195,22 +194,22 @@ io.confluent:kafka-schema-registry-client:5.3.2=default,deploy_jar,runtimeClassp
io.dropwizard.metrics:metrics-core:3.1.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.github.classgraph:classgraph:4.8.104=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.github.java-diff-utils:java-diff-utils:4.12=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-alts:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-api:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-auth:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-alts:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-api:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-auth:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-census:1.49.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-context:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-core:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-googleapis:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-grpclb:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-netty-shaded:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-context:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-core:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-googleapis:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-grpclb:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-netty-shaded:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-netty:1.49.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-protobuf-lite:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-protobuf:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-protobuf-lite:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-protobuf:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-rls:1.49.2=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-services:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-stub:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-xds:1.51.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-services:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-stub:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.grpc:grpc-xds:1.51.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.netty:netty-buffer:4.1.77.Final=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.netty:netty-codec-http2:4.1.77.Final=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
io.netty:netty-codec-http:4.1.77.Final=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -278,8 +277,8 @@ org.apache.beam:beam-vendor-guava-26_0-jre:0.1=default,deploy_jar,runtimeClasspa
org.apache.commons:commons-compress:1.22=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
org.apache.commons:commons-csv:1.9.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.apache.commons:commons-lang3:3.12.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.apache.httpcomponents:httpclient:4.5.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.apache.httpcomponents:httpcore:4.4.15=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.apache.httpcomponents:httpclient:4.5.14=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.apache.httpcomponents:httpcore:4.4.16=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath
org.bouncycastle:bcpg-jdk15on:1.67=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.bouncycastle:bcpkix-jdk15on:1.67=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -290,7 +289,7 @@ org.checkerframework:checker-qual:2.11.1=checkstyle
org.checkerframework:checker-qual:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
org.checkerframework:checker-qual:3.12.0=compileClasspath
org.checkerframework:checker-qual:3.13.0=testCompileClasspath
org.checkerframework:checker-qual:3.28.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.checkerframework:checker-qual:3.29.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.checkerframework:dataflow:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
org.checkerframework:javacutil:3.0.0=annotationProcessor,errorprone,testAnnotationProcessor
org.codehaus.jackson:jackson-core-asl:1.9.13=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -305,7 +304,7 @@ org.eclipse.jetty:jetty-server:9.4.49.v20220914=default,deploy_jar,runtimeClassp
org.eclipse.jetty:jetty-servlet:9.4.49.v20220914=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.eclipse.jetty:jetty-util-ajax:9.4.49.v20220914=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.eclipse.jetty:jetty-util:9.4.49.v20220914=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.flywaydb:flyway-core:9.10.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.flywaydb:flyway-core:9.11.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.glassfish.jaxb:jaxb-runtime:2.3.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.glassfish.jaxb:txw2:2.3.1=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.gwtproject:gwt-user:2.10.0=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -326,15 +325,15 @@ org.jetbrains:annotations:17.0.0=default,deploy_jar,runtimeClasspath,testCompile
org.joda:joda-money:1.0.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.json:json:20200518=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.jsoup:jsoup:1.15.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-api:5.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-engine:5.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-commons:1.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-engine:1.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-launcher:1.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-runner:1.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-suite-api:1.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-suite-commons:1.9.1=testRuntimeClasspath
org.junit:junit-bom:5.9.1=testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-api:5.9.2=testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-engine:5.9.2=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-commons:1.9.2=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-engine:1.9.2=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-launcher:1.9.2=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-runner:1.9.2=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-suite-api:1.9.2=testCompileClasspath,testRuntimeClasspath
org.junit.platform:junit-platform-suite-commons:1.9.2=testRuntimeClasspath
org.junit:junit-bom:5.9.2=testCompileClasspath,testRuntimeClasspath
org.jvnet.staxex:stax-ex:1.8=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.opentest4j:opentest4j:1.2.0=testCompileClasspath,testRuntimeClasspath
org.ow2.asm:asm-analysis:8.0.1=jacocoAnt
@ -355,8 +354,8 @@ org.postgresql:postgresql:42.5.1=default,deploy_jar,runtimeClasspath,testRuntime
org.reflections:reflections:0.9.12=checkstyle
org.rnorth.duct-tape:duct-tape:1.0.8=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
org.slf4j:slf4j-api:1.7.36=testCompileClasspath
org.slf4j:slf4j-api:2.0.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.slf4j:slf4j-jdk14:2.0.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.slf4j:slf4j-api:2.0.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.slf4j:slf4j-jdk14:2.0.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.springframework:spring-core:5.3.18=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.springframework:spring-expression:5.3.18=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.springframework:spring-jcl:5.3.18=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
@ -365,7 +364,7 @@ org.testcontainers:jdbc:1.17.6=default,deploy_jar,runtimeClasspath,testRuntimeCl
org.testcontainers:junit-jupiter:1.17.6=testCompileClasspath,testRuntimeClasspath
org.testcontainers:postgresql:1.17.6=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.testcontainers:testcontainers:1.17.6=default,deploy_jar,runtimeClasspath,testCompileClasspath,testRuntimeClasspath
org.threeten:threetenbp:1.6.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.threeten:threetenbp:1.6.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.tukaani:xz:1.5=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.w3c.css:sac:1.3=default,deploy_jar,runtimeClasspath,testRuntimeClasspath
org.xerial.snappy:snappy-java:1.1.8.4=default,deploy_jar,runtimeClasspath,testRuntimeClasspath

12
docs/gradle.md
View file

@ -40,12 +40,12 @@ is easier to exclude the suite classes than individual test classes. This is the
reason why all test tasks in the :core project contain the exclude pattern
'"**/*TestCase.*", "**/*TestSuite.*"'
Many Nomulus tests are not hermetic: they modify global state (e.g., the shared
local instance of Datastore) but do not clean up on completion. This becomes a
problem with Gradle. In the beginning we forced Gradle to run every test class
in a new process, and incurred heavy overheads. Since then, we have fixed some
tests, and manged to divide all tests into three suites that do not have
intra-suite conflicts. We will revisit the remaining tests soon.
Many Nomulus tests are not hermetic: they modify global state, but do not clean
up on completion. This becomes a problem with Gradle. In the beginning we forced
Gradle to run every test class in a new process, and incurred heavy overheads.
Since then, we have fixed some tests, and manged to divide all tests into three
suites that do not have intra-suite conflicts. We will revisit the remaining
tests soon.
Note that it is unclear if all conflicting tests have been identified. More may
be exposed if test execution order changes, e.g., when new tests are added or

33
docs/operational-procedures.md
View file

@ -86,35 +86,14 @@ agreement][registry-agreement].
Nomulus provides [code to generate and send these
deposits](./operational-procedures/rde-deposits.md).
### Monthly registry activity reporting
### Monthly registry activity and transaction reporting
ICANN requires monthly activity reporting. The details are contained in
Specification 3 of the [registry agreement][registry-agreement].
ICANN requires monthly activity and transaction reporting. The details are
contained in Specification 3 of the [registry agreement][registry-agreement].
These reports are generated by querying tables of Datastore snapshots which have
been exported to the BigQuery data warehouse. The Nomulus release includes the
queries used by Google. The script is Google-specific at the moment, but should
be easy to modify to support a different instance of Nomulus.
The SQL queries which assemble the report are generated by the Python script
[icann_report_query_builder.py](https://github.com/google/nomulus/blob/master/python/google/registry/reporting/icann_report_query_builder.py).
The script generates a monolithic SQL query which is then executed against the
BigQuery tables to generate a result table containing activity report data for
all the TLDs in the system. With a little work, the data can be extracted into
CSVs in the proper format.
The test script
[icann_report_query_builder_test.py](https://github.com/google/nomulus/blob/master/python/google/registry/reporting/icann_report_query_builder_test.py)
shows how the script is run, and the test data file
[golden_activity_query.sql](https://github.com/google/nomulus/blob/master/python/google/registry/reporting/testdata/golden_activity_query.sql)
shows what the resulting SQL looks like.
### Monthly transaction reporting
ICANN requires monthly transaction reporting. The details are also contained in
Specification 3 of the [registry agreement][registry-agreement].
Transaction reporting is not currently part of the Nomulus release.
These reports are mostly generated by querying the Cloud SQL database. There is
currently a Google proprietary class to query DNS related activities that is
not included in the open source Nomulus release.
### Zone File Access (ZFA)

21
docs/operational-procedures/premium-list-management.md
View file

@ -4,12 +4,12 @@
Nomulus comes with a `StaticPremiumListPricingEngine` that determines premium
prices of domain labels (i.e. the part of the domain name without the TLD) by
checking for their presence on a list of prices in Datastore. `nomulus` is used
to load and update these lists from flat text files. The format of this list is
simple: It is a newline-delimited CSV text file with each line containing the
label and its price (including currency specifier in ISO-4217 format). Any
individual label may not appear more than once in the file. Here's an example of
the formatting:
checking for their presence on a list of prices in the database. `nomulus` is
used to load and update these lists from flat text files. The format of this
list is simple: It is a newline-delimited CSV text file with each line
containing the label and its price (including currency specifier in ISO-4217
format). Any individual label may not appear more than once in the file. Here's
an example of the formatting:
```
premium,USD 100
@ -28,7 +28,7 @@ location that is accessible to all members of your team (ideally in a source
control system for revision tracking). These files should be thought of as the
canonical versions of your premium lists. Note that there is no provided way to
reconstruct a premium list .txt file from the premium list that is loaded into
Datastore (though in principle it would be easy to do by writing a tool to do
the database (though in principle it would be easy to do by writing a tool to do
so), so don't lose those .txt files.
The nomulus repository contains an
@ -37,7 +37,7 @@ The nomulus repository contains an
## Creating a premium list
Once the file containing the premium prices is ready, run the
`create_premium_list` command to load it into Datastore as follows:
`create_premium_list` command to load it into the database as follows:
```shell
$ nomulus -e {ENVIRONMENT} create_premium_list -n exampletld -i exampletld.txt
@ -106,8 +106,9 @@ premiumList=Key<?>(EntityGroupRoot("cross-tld")/PremiumList("exampletld"))
## Listing all available premium lists
The `list_premium_lists` command is used to list all premium lists in Datastore.
It takes no arguments and displays a simple list of premium lists as follows:
The `list_premium_lists` command is used to list all premium lists in the
database. It takes no arguments and displays a simple list of premium lists as
follows:
```shell
$ nomulus -e {ENVIRONMENT} list_premium_lists

2
docs/operational-procedures/rde-deposits.md
View file

@ -26,7 +26,7 @@ need to implement a working and secure key store from which RDE can pull the
private key used to transmit the deposits via sFTP.
For each phase and TLD in the process, the system maintains a `Cursor` entity in
Datastore, which contains a timestamp indicating that everything before the
the database, which contains a timestamp indicating that everything before the
timestamp is current (except for RDE_UPLOAD_SFTP, which works a little
differently). Only if the current time is after the cursor time do the actions
check to see if they have work to do. For RDE, there are separate cursor types

6
docs/operational-procedures/reserved-list-management.md
View file

@ -82,7 +82,7 @@ reserved lists (see subsequent sections). The two naming patterns are thus:
## Creating a reserved list
Once the file containing the list of reserved terms is created, run the
`create_reserved_list` command to load it into Datastore as follows. For the
`create_reserved_list` command to load it into the database as follows. For the
purposes of this example, we are creating a common reserved list named
"common_bad-words".
@ -163,8 +163,8 @@ reservedLists=[Key<?>(EntityGroupRoot("cross-tld")/ReservedList("common_bad-word
## Listing all available reserved lists
The `list_reserved_lists` command is used to list all reserved lists in
Datastore. It takes no arguments and displays a simple list of reserved lists in
The `list_reserved_lists` command is used to list all reserved lists in the
database. It takes no arguments and displays a simple list of reserved lists in
newline-delimited format as follows:
```shell