Commit graph

174 commits

Author SHA1 Message Date
Michael Muller
8099789012
Safely lazy load claims and reserved lists (#1177)
* Safely lazy load claims and reserved lists

This moves the entries of all of these lists into "insignificant" fields and
manages them explicitly.

* Additional fixes

Fix a few problems that came up in the merge or weren't caught in earlier
local test runs.

* Changes for review

- removed debug code
- added comments
- improved some methods that were loading the entire claims list
  unnecessarily.

* Fixed javadoc links

* Reformatted

* Minor fix for review
2021-05-25 11:28:30 -04:00
Michael Muller
f7dca7fa96
Make PremiumList.labelsToPrices "insignificant" (#1167)
* Make PremiumList.labelsToPrices "insignificant"

Add the ImmutableObject.Insignificant annotation to labelsToPrices and also
mark it as Transient.  In order to do lazy-loads on this field, we need to do
so explicitly: doing otherwise breaks the immutability contract and prevents
detaching the object upon load.

Note that this is an expedient solution to this problem, but not the optimal
one.  Ideally, the disassociation between PremiumList and its PremiumEntry's
would be more explicit.  However, breaking labelsToPrices out would at minimum
require reworking the Create/UpdatePremiumList commands, which currently rely
on passing around a self-contained PremiumList object, both from the parser
interfaces and to the database.

If this approach is acceptable, we can apply it to ReservedList and ClaimsList
as well (though it may be easier to break the association in those cases).

* Fix premium list "delete" to support a test

* Fix a few more tests

* Changes for review (updated javadocs)

* Minor fixes

* Updated getLablesToPrices() comment

* Format fixes, fixed PremiumEntry interfaces

PremiumEntry can now be SQL only.
2021-05-20 11:21:37 -04:00
gbrodman
a7e8ae5a2c
Add loadOnlyOf method to tm() (#1162)
* Add loadOnlyOf method to tm()

In addition there's a bit of a refator of SqlReplayCheckpoint to make it
more in line with the other singletons. This method is useful for the
singleton classes where we expect at most one entity to exist, e.g.
ServerSecret.
2021-05-20 10:59:01 -04:00
Lai Jiang
10757863ce
Reorder steps (#1159) 2021-05-13 13:15:46 -04:00
Lai Jiang
4246e7e4e0
Add indexes on contacts in the Domain table (#1145)
These indexes are used to find if a contact is linked to a domain in
during a contact delete.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1145)
<!-- Reviewable:end -->
2021-05-13 10:47:35 -04:00
gbrodman
0d67ea3a6e
Combine the two Lock classes into one class (#1126)
* Combine the two Lock classes into one class

This allows us to remove the DAO and to just treat locks the same as we
would treat any other object -- generically grabbing them from the
transaction manager.

We do not need to be concerned about the changeover between Datastore
and SQL because we assume that any such changeover will require
sufficient downtime that any currently-valid acquired locks will expire
during the downtime. Otherwise, we could get into a situation where an
action has acquired a particular lock in Datastore but not SQL.
2021-05-11 16:37:40 -04:00
Lai Jiang
8d18450e56
Update README.md (#1146)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1146)
<!-- Reviewable:end -->
2021-05-11 13:40:07 -04:00
gbrodman
2c4c0bf9f8
Convert more tests to use @DualDatabaseTest and SQL in general (#1101)
Nothing super crazy here other than persisting the entity changes in
DomainDeleteFlow at the end of the flow rather than almost at the end.
This means that when we return the results we give the results as they
were originally present, rather than the subsequently-changed values.
2021-04-23 18:26:44 -04:00
gbrodman
5d3e9da750
Defer all foreign keys in SQL (#1094)
* Defer all foreign keys in SQL

The main difference here is that the constraint violation exceptions
won't be thrown until the transaction is completed, rather than when the
insert is first performed within the transaction. We get the same error
message either way. The primary benefit to this is that when dealing
with large operations inside a single transaction (flows), we don't need
to worry about the order of insertions of removals with regards to
foreign keys.
2021-04-21 14:29:20 -04:00
gbrodman
23896b64c7
Set default value of 1 for new not-null columns (#1097)
Use 1 since it's the constant singleton ID
2021-04-20 15:25:20 -04:00
Weimin Yu
8b41b5c76f
Upgrade testcontainers to work around a race (#1080)
* Upgrade testcontainers to work around a race

testcontainers 1.15.? has a race condition that occassionally causes deadlocks.
This can be worked around by upgrading to 1.15.2 and set transport type to
http5.

See https://github.com/testcontainers/testcontainers-java/issues/3531
for more information.

There are two changes that are not lockfiles:
- dependencies.gradle
- java_common.gradle
2021-04-14 09:45:09 -04:00
gbrodman
d35460f14c
Convert TmchCrl and ServerSecret to cleaner tm() impls (#1068)
* Convert TmchCrl and ServerSecret to cleaner tm() impls

When I implemented this originally I knew a lot less than I know now
about how we'll be storing and retrieving these singletons from SQL. The
optimal way here is to use the single SINGLETON_ID as the primary key,
that way we always know how to create the key that we can use in the
tm() retrieval.

This allows us to use generic tm() methods and to remove the handcrafted
SQL queries.
2021-04-13 20:50:07 -04:00
gbrodman
92dcacf78c
Add a beforeSqlSave callback to ReplaySpecializer (#1062)
* Add a beforeSqlSave callback to ReplaySpecializer

When in the Datastore-primary and SQL-secondary stage, we will want to
save the EppResource-at-this-point-in-time field in the *History
objects so that later on we can examine the *History objects to see what
the resource looked like at that point in time.

Without this PR, the full object at that point in time would be lost
during the asynchronous replay since Datastore doesn't know about it.

In addition, we modify the HistoryEntry weight / priority so that
additions to it come after the additions to the resource off of which it
is based. As a result, we need to DEFER some foreign keys so that we can
write the billing / poll message objects before the history object that
they're referencing.
2021-04-12 12:11:20 -04:00
Weimin Yu
9dd08c48bc
Use credential in secretmanager to deploy schema (#1055)
* Use credential in secretmanager to deploy schema

Fetch the schema_deployer credential from SecretManager when deploying
the schema to Cloud SQL.
2021-04-06 09:43:15 -04:00
gbrodman
a4e078305d
Embed a ZonedDateTime as the UpdateAutoTimestamp in SQL (#1033)
* Embed a ZonedDateTime as the UpdateAutoTimestamp in SQL

This means we can get rid of the converter and more importantly, means
that reading the object from SQL does not affect the last-read time (the
test added to UpdateAutoTimestampTest failed prior to the production
code change).

For now we keep both time fields in UpdateAutoTimestamp however
post-migration, we can remove the joda-time field if we wish.

Note: I'm not sure why <now> is the time that we started getting
LazyInitializationExceptions in the LegacyHistoryObject and
ReplayExtension tests but we can solve that by just examining /
initializing the object within the transaction.
2021-03-29 11:59:08 -04:00
Michael Muller
de09994b36
Add replay to remaining (non-trivial) flow tests (#1020)
* Add replay to remaining (non-trivial) flow tests

Convert all remaining flow tests to do replay/compare testing.  In the course
of this:
- Move the class specific SetClock extension into its own place.
- Fix another "cyclic" foreign key (there may be another solution in this case
  because HostHistory is actually different from HistoryEntry, but that would
  require changing the way we establish priority since HostHistory is not
  distinguished from HistoryEntry in the current methodology)
2021-03-19 13:20:53 -04:00
Weimin Yu
6bee440194
Update creation script for schema_deployer (#1019)
* Update creation script for schema_deployer

Move the create user command for schema_deployer before the
initialization of roles. As the owner of all schema objects, it needs to
be present before grant statements are executed.

Also fixed a bug in credential printing, which fails when the password
contains '%'.
2021-03-18 22:24:03 -04:00
gbrodman
8b2ddf8249
Refactor Cursor to exist in one class (#988)
This allows us to get rid of the DAO as well as the sanity-checking
methods since we can be reasonably sure that the fields will be the
same. Future PRs will add conversions from ofy() to tm() calls that will
make sure that we get the same proper data in both Datastore and SQL
2021-03-18 21:58:07 -04:00
Michael Muller
6bc943bb7d
Convert more flow tests to replay/compare (#1009)
* Convert more flow tests to replay/compare

Add the replay extension to another batch of flow tests.  In the course of
this:

- Refactor out domain deletion code into DatabaseHelper so that it can be used
  from multiple tests.
- Make null handling uniform for contact phone numbers.

* Convert postLoad method to onLoad.

* Remove "Test" import missed during rebase

* Deal with persistence of billing cancellations

Deal with the persistence of billing cancellations, which were added in the
master branch since before this PR was initially sent for review.

* Adding forgotten flyway file

* Removed debug variable
2021-03-18 14:31:58 -04:00
Weimin Yu
deb84cf74d
Add schema_deployer SQL user to SecretManager (#1018)
* Add schema_deployer SQL user to SecretManager

Add the 'schema_deployer' user to the SecretManager so that its
credential can be set up. The schema deployment process will use this
user instead of the 'postgres' user.

Changed the output of the get_sql_credential command for the schema
deployment process.

Added a sql script that documents the privileges granted to
'schema_deployer'.
2021-03-17 19:31:44 -04:00
Michael Muller
7b673c90db
Use ReplaySpecializer to fix DomainBase replays (#991)
* Use ReplaySpecializer to fix DomainBase replays

DomainBase currently has a number of ancillary objects that require a
cascading delete that doesn't get propagated.  Implement beforeSqlDelete() in
DomainContent to delete these child entities.

* Remove unnecessary Query variable

* Fix rebase error
2021-03-10 17:24:23 -05:00
Ben McIlwain
e5801e1b60
Update more dependencies to newer versions (#999)
* Update more dependencies to newer versions

* Add lockfiles and back out 2 problematic dep updates

* Fix the build (backs out more changes)

* Back out qdox 2.0 too
2021-03-10 11:22:07 -05:00
Ben McIlwain
a52a8695e3
Update a lot of Cloud API dependencies (#994)
Hopefully this will help us to resolve our 'gRPC is too old' warning (gRPC isn't
a direct dependency so it's likely coming in through these).
2021-03-09 13:24:41 -05:00
Weimin Yu
1f4cf5bdb6
Fix flyway invocation in Gradle script (#993)
* Fix flyway invocation in Gradle script

Script wrongly assumed that Flyway task is invoked if --environment is
set.

Bug was introduced in go/r3pr/940
2021-03-08 13:59:28 -05:00
Weimin Yu
63f8fcef18
Clean up Gradle Flyway tasks in :db (#990)
* Clean up Gradle Flyway tasks in :db

Simplified the command line by revising the semantics of some
properties.

Added examples of Flyway task invocations.

This script still uses the GCS file-based credential. We will migrate it
to the Secret Manager soon.
2021-03-04 19:58:09 -05:00
gbrodman
31bf4bd76d
Add a third poll message column in TransferData for domain transfers (#974)
Because we don't store serverApproveEntities specifically as a set in
the SQL world, we need to make sure that the entities are all separated
and stored if they exist. For domain transfers, there exist three
separate poll messages (client losing, client gaining, autorenew) so we
need to store and retrieve that one.

Founnd this while converting domain transfer flows to SQL.
2021-02-25 17:22:53 -05:00
Weimin Yu
ffe3124ee1
Properly set up JPA in BEAM workers (#965)
* Properly set up JPA in BEAM workers

Sets up a singleton JpaTransactionManger on each worker JVM for all
pipeline nodes to share.

Also added/updated relevant dependencies. The BEAM SDK version change
caused the InitSqlPipeline's graph to change.
2021-02-22 18:11:32 -05:00
Lai Jiang
0e74df82df
Update testcontainers to 1.51.1 (#958)
Per release note
(https://github.com/testcontainers/testcontainers-java/releases/tag/1.15.1),
this version is needed to support docker version 20.10 and above.
2021-02-08 14:31:17 -05:00
Lai Jiang
b59d2ae419
Upgrade to Gradle 6.8.2 (#957) 2021-02-08 09:35:43 -05:00
Shicong Huang
48de5d8375
Convert ofy() to tm() for all contact transfer flows (#937)
* Convert ofy() to tm() for all contact transfer flows

* Resolve comments
2021-01-22 09:38:51 -05:00
Shicong Huang
56e384aa4f
Restore symmetric VKey referenced by BillingEvent.Cancellation (#928) 2021-01-14 13:16:24 -05:00
Weimin Yu
cb63c3dd80
Add unique constraints on domain_hosts (#911)
* Add unique constraints on domain_hosts

Add unique constraints on DomainHost (child of DomainBase) and
DomainHistoryHost (child of DomainHistory). DomainHost is non-entity
embedded object and Hibernate does not define indexes automatically.

This should improve read and write performance of the parent entities.
2020-12-21 18:22:24 -05:00
Shicong Huang
6e2bbd1a7e
Add BillingVKey to restore symmetric VKey in GracePeriodBase (#902)
* Use PollMessageVKey to replace VKey<PollMessage> in DomainBase

* Revert changes to DomainContent

* Use BillingVKey in GracePeriodBase to restore symmetric vkey

* Rebase on HEAD
2020-12-17 14:13:47 -05:00
gbrodman
cb16a7649f
Add a scrap command to backfill Spec11 threats (#897)
This parses through all pre-existing Spec11 files in GCS (starting at
2019-01-01 which is basically when the new format started) and maps them
to the new Spec11ThreatMatch objects.

Because the old format stored domain names only and the new format stores
names + repo IDs, we need to retrieve the DomainBase objects from the
point in time of the scan (failing if they don't exist). Because the
same domains appear multiple times (we estimate a total of 100k+ entries
but only 1-2k unique domains) we cache the DomainBase objects that we
retrieve from Datastore.
2020-12-15 16:18:27 -05:00
Michael Muller
64e1a4b345
Make Domain -> BillingEvent FK deferred (#890)
* Make Domain -> BillingEvent FK deferred

It appears that Hibernate can sporadically introduce FK constraint failures
when updating a Domain to reference a new BillingEvent and then deleting the
old BillingEvent, causing a flakey test failure in DomainDeleteFlowTest.  This
may be due to the fact that this FK relationships is not known to hibernate.

An alternate solution appears to be to flush after every update, but that
likely has some pretty serious performance implications.
2020-11-30 18:06:07 -05:00
Michael Muller
cde1c78f5e
Add replay-testing to DomainDeleteFlowTest (#886)
* Minor fixes:

- Initialize "requestedByRegistrar" to false (it's non-nullable).
- Store test entities (registrar, hosts and contacts) in JPA.

* Flyway changes

* Add ReplayExtension to DomainDeleteFlowTest

* Check in latest ER diagrams
2020-11-25 11:15:10 -05:00
Weimin Yu
e1ce357904
Drop foreign key constraints on PollMessages (#875)
* Drop foreign key constraints on PollMessages

PollMessages are deleted after being acked by recipients.
Other tables cannot have foreign key constraints on them.
2020-11-18 20:28:56 -05:00
Shicong Huang
1d91a8e647
Use DomainHistoryVKey to restore symmetric VKey (#874)
* Use DomainHistoryVKey to restore symmetric VKey

* Rebase on HEAD
2020-11-17 16:32:32 -05:00
Michael Muller
ab7ee51fb2
Add an extension to verify transaction replay (#857)
* Add an extension to verify transaction replay

Add ReplayExtension, which can be applied to test suites to verify that
transactions committed to datastore can be replayed to SQL.

This introduces a ReplayQueue class, which serves as a stand-in for the
current lack of replay-from-commit-logs.  It also includes replay logic in
TransactionInfo which introduces the concept of "entity class weights."
Entity weighting allows us store and delete objects in an order that is
consistent with the direction of foreign key and deferred foreign key
relationships.  As a general rule, lower weight classes must have no direct or
indirect non-deferred foreign key relationships on higher weight classes.

It is expected that much of this code will change when the final replay
mechanism is implemented.

* Minor fixes:

- Initialize "requestedByRegistrar" to false (it's non-nullable). [reverted
  during rebase: non-nullable was removed in another PR]
- Store test entities (registrar, hosts and contacts) in JPA.

* Make testbed save replay

This changes the replay system to make datastore saves initiated from the
testbed (as opposed to just the tested code) replay when the ReplayExtension
is enabled.  This requires modifications to DatastoreHelper and the
AppEngineExtension that the ReplayExtension can plug into.

This changes also has some necessary fixes to objects that are persisted by
the testbed (such as PremiumList).
2020-11-17 13:29:50 -05:00
Weimin Yu
51942fcaad
Make some columns nullable in History tables (#873)
* Make some columns nullable in History tables

xmlBytes is made nullable in all history tables since changes performed
by backend actions would not have it. In addition, epp requests are not saved to
ContactHistory since data may contain PII.

requestedByRegistrar in all history tables are made nullable. This
property is set from metadata in epp requests. Null means not provided.
2020-11-13 15:25:54 -05:00
Weimin Yu
ae6b414b82
Add a Secret Manager client for Nomulus (#872)
* Add a Secret Manager client for Nomulus
2020-11-12 17:12:52 -05:00
Shicong Huang
de20334a66
Add SQL schema for GracePeriodHistory (#746)
* Add schema for GracePeriodHistory

Rebase on HEAD

Rebase on HEAD

Rebase on HEAD and rename column

Use OfyService to generate id

Refactor GracePeriodsSubject

Rebase on HEAD

Remove GracePeriodSubject and GracePeriodsSubject

Rebase on HEAD

Rebase on HEAD

Rebase on HEAD

Add gracePeriodHistoryRevisionId and remove some foreign key

* Rebase on HEAD
2020-11-12 10:06:58 -05:00
gbrodman
f1eb1a8fe7
Add SQL replay checkpoint object to SQL (#868)
* Add SQL replay checkpoint object to Datastore

This will be part of the asynchronous commit-log replay to SQL. Whenever
we successfully export commits up to a particular time, we should
persist that time so we don't replay the same commits again (it is not
idempotent)

* Move SqlReplayCheckpoint from DS to SQL

* Responses to CR
2020-11-10 17:09:18 -05:00
gbrodman
0c6363c04f
Persist two singleton entities in SQL tables (#860)
* Persist two singleton entities in SQL tables

A table might not be the best place to store singleton entities, but by
doing this we ensure we can easily inspect them later and use the same
sort of persistence logic for these that we do elsewhere.

ServerSecret is stored upon retrieval so that we make sure that the same
secret is used in both Datastore and SQL (we wouldn't want to change
it).

* Responses to CR

* Don't have a separate ID for the singleton entities

* Rename secret UUID

* Rename and regenerate
2020-11-09 13:47:42 -05:00
Shicong Huang
420f3bf380
Add missing foreign key constraints (#845)
* Add missing foreign key constraints

* Fix failed unit tests
2020-11-09 10:55:29 -05:00
Shicong Huang
30d57d9476
Upgrade Postgres realted packages to latest version (#861) 2020-11-06 10:14:44 -05:00
gbrodman
d685f7e2df
Add a SQL schema and DAO for KmsSecretRevision (#840)
* Add a SQL schema and DAO for KmsSecretRevision

The dual-object nature of KmsSecret and KmsSecretRevision will not be
necessary once we have moved to SQL. In that world, the only object will
be the one now called KmsSecretRevision. KmsSecretRevision already
stores its parent so all we need to do is convert that key to the String
secretName (or from the secretName to the key, if loading from SQL) and
select the max revision ID for a given secret name.

In a future PR, we will add a dual-writing DAO to these objects and
perform the dual writes, similar to how ReservedList functions.

* Regenerate diagram

* Rename revisionId and cryptoKeyVersionName

* Fix SQL files and diagram
2020-10-30 18:45:43 -04:00
gbrodman
40eef2a06c
Add SQL schema and DAO for SignedMarkRevocationList (#850)
* Add SQL schema and DAO for SignedMarkRevocationList

This gets saved every day so we're not concerned about history, meaning
we can dual-write and/or dual-read without concern. The structure here
is somewhat similar to the ClaimsListDao and related classes.

* Update the DB files
2020-10-30 17:52:09 -04:00
Shicong Huang
5bba65835a
Change primary key of DelegationSignerData and add its history table (#841)
* Change primary key of DelegationSignerData and add its history table

* Change primary key and resolve comments

* Rebase on HEAD
2020-10-29 16:19:15 -04:00
Shicong Huang
1e51f51979
Use TransactionManager APIs in DatastoreHelper (#849)
* Make DatastoreHelper support Postgresql

* Rebase on HEAD

* Resolve comments

* Use put* inside insert* and update*

* Resolve comments
2020-10-29 11:41:04 -04:00