Commit graph

3561 commits

Author SHA1 Message Date
sarahcaseybot
0b6a5aec96 Modify DeleteReservedListCommand to delete from both databases (#1025)
* Modify DeleteReservedListCommand to use both databases

* switch to confirming command

* fix typo
2021-03-19 18:49:15 -04:00
Ben McIlwain
4ed23f6813 Add a jpaTm().query(...) convenience method (#1023)
* Add a jpaTm().query(...) convenience method

This replaces the more ungainly jpaTm().getEntityManager().createQuery(...).

Note that this is in JpaTransactionManager, not the parent TransactionManager,
because this is not an operation that Datastore can support. Once we finish
migrating away from Datastore this won't matter anyway because
JpaTransactionManager will be merged into TransactionManager and then deleted.

In the process of writing this PR I discovered several other methods available
on the EntityManager that may merit their own convenience methods if we start
using them enough. The more commonly used ones will be addressed in subsequent
PRs. They are:

jpaTm().getEntityManager().getMetamodel().entity(...).getName()
jpaTm().getEntityManager().getCriteriaBuilder().createQuery(...)
jpaTm().getEntityManager().createNativeQuery(...)
jpaTm().getEntityManager().find(...)

This PR also addresses some existing callsites that were calling
getEntityManager() rather than using extant convenience methods, such as
jpa().insert(...).
2021-03-19 16:34:37 -04:00
Michael Muller
c827ea033a Add replay to remaining (non-trivial) flow tests (#1020)
* Add replay to remaining (non-trivial) flow tests

Convert all remaining flow tests to do replay/compare testing.  In the course
of this:
- Move the class specific SetClock extension into its own place.
- Fix another "cyclic" foreign key (there may be another solution in this case
  because HostHistory is actually different from HistoryEntry, but that would
  require changing the way we establish priority since HostHistory is not
  distinguished from HistoryEntry in the current methodology)
2021-03-19 13:20:53 -04:00
Weimin Yu
5befc142f5 Attempt to fix flakey deleteExpiredDomain test (#1022)
* Attempt to fix flakey deleteExpiredDomain test

Though hard to reproduce locally, the test_deletesThreeDomainsInOneRun
test has failed multiple times on Kokoro. The root cause may be the
non-transactional query executed by the Action object, which was by
design. Observing that the other test never fails, this PR follows its behavior
and adds a transactional query before invoking the action.
2021-03-19 12:38:54 -04:00
Weimin Yu
21655bc539 Allow nom_build to run in Cloudbuild (#1021)
* Allow nom_build to run in Cloudbuild

Our builder comes with python3.6 and cannot support nom_build out of
box. Nom_build requires dataclasses which is introduced in v3.7.

I haven't found an easy way to get python3.7+ without changing the base
linux image. This PR explicitly installs dataclasses.
2021-03-19 11:28:18 -04:00
gbrodman
8a1200dc4c Create a ClaimsListDualDatabaseDao (#1011)
The dual DAO takes care of switching between databases, comparing the
results of one to the results of the other, and caching the result. All
calls to ClaimsList retrieval or storing should use the
dual-database-DAO.

Previously, calls to comparing the lists were somewhat scattered
throughout the codebase. Now, there is one class for retrieval and
comparison (the dual DAO), one class for retrieval from SQL (the SQL
DAO), and one class for retrieval from Datastore (ClaimsListShard
itself, though the retrieval could be moved in to a separate DAO if we
wished).

In addition, we rename the ClaimsListDao to ClaimsListSqlDao
2021-03-18 23:37:08 -04:00
Weimin Yu
5e4336fd3b Update creation script for schema_deployer (#1019)
* Update creation script for schema_deployer

Move the create user command for schema_deployer before the
initialization of roles. As the owner of all schema objects, it needs to
be present before grant statements are executed.

Also fixed a bug in credential printing, which fails when the password
contains '%'.
2021-03-18 22:24:03 -04:00
gbrodman
a0c710ab18 Refactor Cursor to exist in one class (#988)
This allows us to get rid of the DAO as well as the sanity-checking
methods since we can be reasonably sure that the fields will be the
same. Future PRs will add conversions from ofy() to tm() calls that will
make sure that we get the same proper data in both Datastore and SQL
2021-03-18 21:58:07 -04:00
Michael Muller
1a0ee97633 Convert more flow tests to replay/compare (#1009)
* Convert more flow tests to replay/compare

Add the replay extension to another batch of flow tests.  In the course of
this:

- Refactor out domain deletion code into DatabaseHelper so that it can be used
  from multiple tests.
- Make null handling uniform for contact phone numbers.

* Convert postLoad method to onLoad.

* Remove "Test" import missed during rebase

* Deal with persistence of billing cancellations

Deal with the persistence of billing cancellations, which were added in the
master branch since before this PR was initially sent for review.

* Adding forgotten flyway file

* Removed debug variable
2021-03-18 14:31:58 -04:00
Weimin Yu
49183db199 Add schema_deployer SQL user to SecretManager (#1018)
* Add schema_deployer SQL user to SecretManager

Add the 'schema_deployer' user to the SecretManager so that its
credential can be set up. The schema deployment process will use this
user instead of the 'postgres' user.

Changed the output of the get_sql_credential command for the schema
deployment process.

Added a sql script that documents the privileges granted to
'schema_deployer'.
2021-03-17 19:31:44 -04:00
Ben McIlwain
1132b29679 Clear autorenew end time when a domain is restored (#1015)
* Clear autorenew end time when a domain is restored

This allows us to still see in the database which now-deleted domains had
reached expiration, while correctly not re-deleting the domain immediately if
the registrar pays to explicitly restore the domain.

This also resolves some TODOs around data migration for this field on domain so
that it's not null, as said migration has already been completed.
2021-03-17 15:39:13 -04:00
Michael Muller
60540fd4bf Allow java-format to use java from the PATH (#1014)
* Allow java-format to use java from the PATH

When invoking java from the google-java-format-git-diff.sh script, if there is
no JAVA_HOME environment variable, attempt to instead run the java binary that
is on the PATH.

This also adds a few checks to verify that a java binary is available in one
of those locations and that the version discovered is Java 11 (which we know
to be compatible with the google-java-format jar).

Tested:
- unset JAVA_HOME, verified that we get the version on the PATH
- Set JAVA_HOME to an invalid directory, verified that we get an error.
- Changed the "which" command to lookup an nonexistent binary, unset JAVA_HOME
  and verified that we get a "java not found" error.
- Changed the path to point to an old version of java, verified that we get a
  "bad java version" error.
- Verified that the script still runs normally.
2021-03-17 10:29:32 -04:00
Ben McIlwain
087dcb071a Remove grace period ID @OnLoads now that migration is complete (#1016)
* Remove grace period ID @OnLoads now that migration is complete

I verified in BigQuery that all grace period IDs are now allocated (as expected
given that the re-save all EPP resource mapreduce has been run several times
since this migration started last year). The query I used for verification is:

SELECT fullyQualifiedDomainName, gp, ot
FROM `domain-registry.latest_datastore_export.DomainBase`
JOIN UNNEST(gracePeriods.billingEventRecurring) AS gp
JOIN UNNEST(gracePeriods.billingEventOneTime) AS ot
WHERE gp.id IS NULL or ot.id IS NULL

BUG=169873747
2021-03-17 10:18:53 -04:00
sarahcaseybot
fe98741275 Refactor ReservedListDualDatabaseDao for easy database cutover (#1003)
* Refactor ReservedListDualDatabaseDao

* Fix merge conflict

* Fix test name

* Fix tests

* more small fixes

* Format fix
2021-03-16 16:37:14 -04:00
Ben McIlwain
8caaba0d84 Add daily cron entries to for DeleteExpiredDomainsAction (#1013)
* Add daily cron entries to for DeleteExpiredDomainsAction

This also requires setting this action to GET instead of POST, as GAE cron makes
GET requests.
2021-03-16 14:57:32 -04:00
Weimin Yu
7c3d0dd1a9 Use shared jar to stage BEAM pipeline if possible (#1008)
* Use shared jar to stage BEAM pipeline if possible

Allow multiple BEAM pipelines with the same classes and dependencies to
share one Uber jar.

Added metadata for BulkDeleteDatastorePipeline.

Updated shell and Cloud Build scripts to stage all pipelines in one
step.
2021-03-16 13:19:30 -04:00
Weimin Yu
24db87a4cf Closing the bug regarding Cloud SQL connection configs (#1012)
* Add comments to Cloud SQL configs

I believe the similarity in trace to https://github.com/brettwooldridge/HikariCP/issues/1212
is misleading.

The real cause of the exceptions may be that we ran out of connections. At the
time, the production Cloud SQL server could handle 500 connections at the
maximum. That number was within reach of a busy Nomulus server.

The maximum connection in production has been increased to 1000. We
haven't encountered this issue for a long time. All connection problems
are due to Cloud SQL maintenance or other GCP related issues.

This issue is tracked by b/154720215, which is being closed with this
PR.
2021-03-16 10:29:30 -04:00
Ben McIlwain
328e9b5c8c Improve logging messages and error level for DeleteExpiredDomainsActions (#1010)
* Improve logging messages and error level for DeleteExpiredDomainsActions
2021-03-15 23:24:32 -04:00
gbrodman
1263993733 Convert DomainTransferRequestFlow to tm() calls (#1002)
* Convert DomainTransferRequestFlow to tm() calls

Besides the standard ofy-to-tm conversions this includes storing the
billing event cancellation VKey in the DomainTransferData object so that
we know to handle it on process / cancellation.
2021-03-15 20:01:59 -04:00
Michael Muller
72f176758d Remove now-unused "logger" from ReplaySpecializer (#1007)
* Remove now-unused "logger" from ReplaySpecializer

This causes a build warning.

* Take out the import, too
2021-03-12 13:26:45 -05:00
Michael Muller
e3ab5ed4c1 Don't use --fork-point to determine merge base (#1001)
* Don't use --fork-point to determine merge base

It turns out that the --fork-point option is subtle and error-prone.  Its
intent is not to show the nearest common base commit, but rather the commit
on a branch that the HEAD (in this case) was originally forked off of,
_whether it is currently part of the history of the specified branch or not_
(this can happen if the branch is rewritten).  The option also relies on the
presence of the fork point in the reflog for the branch, which can be
discarded in the course of a "git gc".

It is fairly easy to construct a case where the use of --fork-point causes an
error and outputs nothing.  In fact, I discovered the problem as a result of
this occuring spontaneously on one of my own branches (likely related to a
rebase).  Since the fork-point is empty, we end up diffing against the index
instead of the common commit.

This may have been a factor in some of the unrelated reformatting that we've
seen in past PRs.

Change this to a simple "merge-base origin/master HEAD", which outputs the
commit id of the most recent common base revision.

This change also quotes the forkPoint variable, which likely would have
resulted in an error in this case instead of silently producing the wrong
output.
2021-03-12 11:08:28 -05:00
Weimin Yu
39816bf7cd Stage the init_sql_pipeline in CloudBuild (#1004)
* Stage the init_sql_pipeline in CloudBuild

Defined metadata file and added Gradle uberJar task for the pipeline,
which are needed for staging.

Updated cloud build script to stage this pipeline during the build
processs.
2021-03-12 10:36:57 -05:00
Weimin Yu
bf5ac3a327 Add TODOs regarding cloud sql database name change (#1005)
* Add TODOs regarding cloud sql database name change

We should choose a different database name for nomulus data since using
'postgres' is bad practice. See b/181693544 more background.

We have decided to delay the db change to the time when we upgrade
postgresql version. This PR adds TODOs to all occurrences of the jdbcUrl
property, including those in the internal-repo. This property will change
when we upgrade, so the TODOs will be noticed.
2021-03-12 09:34:13 -05:00
sarahcaseybot
1d9f67ac70 Change the default public TLS enforcement start date (#1006) 2021-03-11 22:27:50 -05:00
Michael Muller
1d01fabb3c Use ReplaySpecializer to fix DomainBase replays (#991)
* Use ReplaySpecializer to fix DomainBase replays

DomainBase currently has a number of ancillary objects that require a
cascading delete that doesn't get propagated.  Implement beforeSqlDelete() in
DomainContent to delete these child entities.

* Remove unnecessary Query variable

* Fix rebase error
2021-03-10 17:24:23 -05:00
sarahcaseybot
bd0c2a2b21 Add ReservedList to CloudSQL in unit tests (#992)
* Add ReservedList to CloudSQL in unit tests

* Remove unnecessary changes

* Change databasehelper to use DualDatabaseDao
2021-03-10 14:16:39 -05:00
Ben McIlwain
8162933fca Update more dependencies to newer versions (#999)
* Update more dependencies to newer versions

* Add lockfiles and back out 2 problematic dep updates

* Fix the build (backs out more changes)

* Back out qdox 2.0 too
2021-03-10 11:22:07 -05:00
Lai Jiang
76b4bd3c3a Print JAVA_HOME and PATH in the java format checker task (#997)
* Print JAVA_HOME and PATH in the java format checker task

This allows is to identify if the java version is incompatible with the
formater checker jar file.

* Show which Java binary is used

* show java version

* Fix exeInBash arguments
2021-03-10 09:00:08 -05:00
Michael Muller
103c87ed7d Print out env variables in java format (#998)
* Print out env variables in java format

Print out JAVA_HOME and PATH variable in the google-java-format-diff.py script
immediately prior to running the underlying java program that does the actual
format checking.

* Use the java binary from JAVA_HOME for java-format

Use "$JAVA_HOME/bin/java" for invoking the java format check instead of
whatever version of java happens to be on the path.

* Removed unused import
2021-03-10 08:55:14 -05:00
Weimin Yu
a5364a5bab Rewrite the JPA output connector for BEAM (#995)
* Rewrite the JPA output connector for BEAM

Following BEAM's IO connector style, added a RegistryJpaIO class to hold
IO connectors, and implemented the Write connector as a static inner
class in it. The JpaTransactionManager used by the Write connector
retrieves SQL credentials from the SecretManager.

Cleaned up SQL-related pipeline parameters.

Converted the InitSqlPipeline to use RegistryJpaIO.
2021-03-09 16:12:04 -05:00
Ben McIlwain
603024d11d Update a lot of Cloud API dependencies (#994)
Hopefully this will help us to resolve our 'gRPC is too old' warning (gRPC isn't
a direct dependency so it's likely coming in through these).
2021-03-09 13:24:41 -05:00
gbrodman
f760327ffd Add SQL queries to RdapDomainSearchAction (#982)
* Add SQL queries to RdapDomainSearchAction

Unfortunately, because ORDER BY uses the locale's sorting functionality,
we end up with some weird sort orders in SQL-land (notably, periods are
ignored / omitted). As a result, a few of the tests have to be separated
out into ofy and SQL versions based on the expected sort order.

In addition, there isn't a way to query @Convert-ed fields in Postgres
via the standard Hibernate / JPA query language, meaning we have to use
a raw Postgres query for that.
2021-03-09 11:11:53 -05:00
Michael Muller
9413dc1e4f Added "show_upgrade_diffs" script (#981)
* Added "show_upgrade_diffs" script

"show_upgrade_diffs" pulls a git directory and a user branch from nomulus and
compares all of the versions of all dependencies specified in all lockfiles in
the master branch with those of the user branch and prints a nice, terse
little colorized report on the differences.

This is useful for reviewing a dependency upgrade.

* Add license header

* Changes requested in review

* Changes for review

- Change format of output so different actions are displayed somewhat
  consistently.
- Make specifying a directory optional, if not specified create a temporary
  directory and clean it up afterwards.
2021-03-09 07:48:06 -05:00
Michael Muller
3327c1ae9b Add a "ReplaySpecializer" to fix certain replays (#989)
* Add a "ReplaySpecializer" to fix certain replays

Due to the fact that a given entity in either database type can map to
multiple entities in the other database, there are certain replication
scenarios that don't quite work.  Current known examples include:

- propagation of cascading deletes from datastore to SQL
- creation of datastore indexed entities for SQL entities (where indexes are a
  first-class concept)

This change introduces a ReplaySpecializer class, which allows us to declare
static method hooks at the entity class level that define any special
operations that need to be performed before or after replaying a mutation for
any given entity type.

Currently, "before SQL delete" is the only supported hook.  A change to
DomainContent demonstrating how this facility can be used to fix problems in
cascading delete propagation will be sent as a subsequent PR.

* Throw exception on beforeSqlDelete failures

* Changes for review
2021-03-09 07:12:15 -05:00
Weimin Yu
1fa57de5ac Fix flyway invocation in Gradle script (#993)
* Fix flyway invocation in Gradle script

Script wrongly assumed that Flyway task is invoked if --environment is
set.

Bug was introduced in go/r3pr/940
2021-03-08 13:59:28 -05:00
gbrodman
f29219cbb7 Convert DomainTransferRejectFlow to use tm() methods (#977)
* Convert DomainTransferRejectFlow to use tm() methods

This change includes a few other necessary dependencies to converting
DomainTransferRejectFlowTest to be a dual-database test. Namely:

- The basic "use tm() instead of ofy()" and "branching database
selection on what were previously raw ofy queries"
- Modification of the PollMessage convertVKey methods to do what they
say they do
- Filling the generic pending / response fields in PollMessage based on what type of
poll message it is (this has to be done because SQL is not very good at
storing ambiguous superclasses)
- Setting the generic pending / repsonse fields in PollMessage upon
build
- Filling out the serverApproveEntities field in DomainTransferData with
all necessary poll messages / billing events that should be cancelled on
rejection
- Scattered changes in DatabaseHelper to make sure that we're saving and
loading entities correctly where we weren't before
2021-03-08 13:24:30 -05:00
Michael Muller
87081e6f65 Disable whois caching in nomulus tool (#980)
* Disable whois caching in nomulus tool

The whois commands previously served output generated from cached EppResource
objects in most cases.  While this is entirely appropriate from server-side,
it is less useful when these commands are run from nomulus tool and, in fact,
when run from the "shell" command this results in changes that have been
applied from the shell not being visible from a subsequent "whois".  The
command may instead serve information on an earlier, cached version of the
resource instead of the latest version.

This implementation uses dagger for parameterization of cached/non-cached
modes.  I did consider the possibility of simply parameterizing the query
commands in all cases as discussed, however, having gone down the
daggerization path and having gotten it to work, I have to say I find this
approach preferrable.  There's really no case for identifying
cached/non-cached on a per-command basis and doing so would require
propagating the flag throughout all levels of the API and all callsites.

Tested: In addition to the new unit test which explicitly verifies the
caching/noncaching behavior of the new commands, tested the actual failing
sequence from "nomulus -e sandbox shell" and verified that the correct results
are served after a mutation.

* Fixed copyright year

* More copyright date fixes

* Added WhoisCommandFactoryTest to fragile tests

I suspect that this test has some contention with other tests, it's not clear
why.
2021-03-08 12:33:23 -05:00
Michael Muller
49d527d4fe Replay Cloud SQL transactions against datastore (#738)
* Replay Cloud SQL transactions against datastore

Implement the ReplicateToDatastore cron job that will apply all Cloud SQL
transactions to the datastore.  The last transaction id is stored in a
LastSqlTransaction entity in datastore.

Note that this will not be activated in production until a) the cron
configuration is updated and b) the cloudSql.replicateTransactions flag is set
to true in the nomulus config file.

* Post-review changes

Fixed immutability issues with LastSqlTransaction, write a single transaction
at a time to datastore.

* Changes requested in review

* Get a batch of SQL transactions

Read a batch of SQL transactions at a time and then process them
transactionally against datastore.

* Bring this up-to-date with the codebase

* Changes requested in review

* Fixed date in copyright
2021-03-05 10:35:30 -05:00
Weimin Yu
db16a2c679 Clean up Gradle Flyway tasks in :db (#990)
* Clean up Gradle Flyway tasks in :db

Simplified the command line by revising the semantics of some
properties.

Added examples of Flyway task invocations.

This script still uses the GCS file-based credential. We will migrate it
to the Secret Manager soon.
2021-03-04 19:58:09 -05:00
Michael Muller
f997f64169 Add replay & compare to more flow tests (#975)
* Add replay & compare to more flow tests

Add replay and comparison testing to another batch of flow tests, apply fixes
as needed.
2021-03-04 08:25:47 -05:00
Lai Jiang
5520329385 Log forbidden HTTP request method at warning (#983)
* Log forbidden HTTP request method at warning

This seems like more reasonable. It will potential issues with how
requests are generated more discoverable in the log.
2021-03-03 21:02:06 -05:00
Weimin Yu
6e10e24d2f Allow AppEngine deployment to qa environment (#986)
* Allow AppEngine deployment to qa environment
2021-03-03 19:31:08 -05:00
sarahcaseybot
af50677612 Reject handshakes with bad TLS protocols and ciphers (#970)
* Reject handshakes with bad TLS protocols and ciphers

* Fix protocols

* make cipher suite list static and fix tests

* Delete unnecessary line

* Add start time configuration for enforcement

* small format fix

* Add multiple ciphersuite test

* fix gradle lint

* fix indentation
2021-03-03 15:47:42 -05:00
Ben McIlwain
7aa7bad986 Actually log the error in DeleteExpiredDomainsAction (#985)
* Actually log the error in DeleteExpiredDomainsAction
2021-03-03 15:21:13 -05:00
Lai Jiang
a3ea1d96b4 Use fanout action to run ICANN report upload job in cron (#984)
GAE cron only issuse HTTP GET requests to the endpoint in question. This
particular only allows POSTs. As a result this cron job never succeeded.
This is not a big problem as this job is meant to catch up any
unforeseen upload failures and in case it needs to catch up but fails,
every month the staging job (which is enqueued corrected by cron) will
eventually catch everything to date.
2021-03-03 09:00:34 -05:00
Lai Jiang
a7f4a885b1 Update a few plugins for Java 11 compatibility (#966)
* Update a few plugins for Java 11 compatibility

Guice 5.0.1 is now compatible with Java 11. However we don't
directly depend on Guice. Rather Soy depends on Guice. So I added a
direct dependency on Guice 5.0 just before Soy in order to frontload Soy
and pull in the newer version.

Mockito 3.7.7 is now compatible with Java 11. The complication is that
we need to use the inline version of Mockito, which among other things
also allows mocking for final classes (hooray!). It will eventually
become the default Mockito mock maker but for now it needs to be
manually activated.

Note that the inline version now introduces another warning:

```
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
```

Which I think is WAI due to how the inline mock maker works. Waiting on
the author to confirm.

After these to changes the only illegal reflective access is caused by
App Engine SDK tools, which we will rid ourselves of when we migrate off
of GAE.

* Restore package-lock.json
2021-03-02 21:01:30 -05:00
gbrodman
25ac7898a8 Add GetPremiumListCommand (#972)
* Add GetPremiumListCommand

When testing the premium list refactor, it would have been nice and
convenient to have this. Currently we have no way of inspecting premium
list contents that I'm aware of.
2021-03-02 19:59:12 -05:00
gbrodman
f78b64d93c Add SQL searching to RdapEntitySearchAction and RdapSearchActionBase (#969)
- Adds a CriteriaQueryBuilder class that allows us to build
CriteriaQuery objects with sane and modular WHERE and ORDER BY clauses.
CriteriaQuery requires that all WHERE and ORDER BY clauses be specified
at the same time (else later ones will overwrite the earlier ones) so in
order to have a proper builder pattern we need to wait to build the
query object until we are done adding clauses.

- In addition, encapsulating the query logic in the CriteriaQueryBuilder
class means that we don't need to deal with the complicated Root/Path
branching, otherwise we'd have to keep track of CriteriaQuery and Root
objects everywhere.

- Added a REPLAYED_ENTITIES TransitionId that will represent all
replayed entities, e.g. EppResources. Also sets this, by default, to
always be CLOUD_SQL if we're using the SQL transaction manager in tests.

- Added branching logic in RdapEntitySearchAction based on that transition
ID that determines whether we do the existing ofy query logic or JPA
logic.
2021-03-02 13:13:55 -05:00
Michael Muller
519cbfb660 Add replay & compare to ContactCreateFlowTest test (#976)
* Add replay & compare to ContactCreateFlowTest test

This test exposed a write priority issue between ContactResource and
ContactHistory entries.
2021-03-02 11:50:03 -05:00
Lai Jiang
179ffeb148 Upgrade to Gradle 6.8.3 (#979)
* Upgrade to Gradle 6.8.3
2021-03-01 21:11:06 -05:00