Commit graph

2966 commits

Author SHA1 Message Date
Ben McIlwain
2a29ecc2a2
Allow claims keys on domain creates during quiet periods (#453)
* Allow claims keys on domain creates during quiet periods

Since we're using TldState.QUIET_PERIOD for the .new LRP (in which only reserved
domains with allocation tokens may be registered), we also need to support
claims keys during this phase. Otherwise, domains that appear in MarksDB will
not be registrable during LRP, even if the correct claims key is provided.

This is based on an error report from a registrar, in which a correct-looking
domain create was failing with the error message "Declared launch extension
phase does not match the current registry phase".
2020-01-23 12:17:14 -05:00
gbrodman
16b4d15292
Allow checking reserved domains with a valid token (#451)
* Add test for checking reserved domain with token

* Allow domain checks with a valid token if the domain isn't fully blocked

* Check specifically for specific-use or anchor reservations

* Add test for LRP in-time token
2020-01-23 11:45:07 -05:00
gbrodman
7df3d85243
Refactor lock/unlock commands to use Registry Locks (#390)
* Refactor lock/unlock commands to use Registry Locks

* CR responses

* Remove unnecessary XML test files

* Add tests

* Respond to CR

* Refactor further the creation and verification of locks

* remove isUnlocked

* Responses to CR

* Fix tests

* Add admin-override back to unlocking

* Add TODO

* Fix imports
2020-01-23 10:22:56 -05:00
Ben McIlwain
02e7106262
Refactor g.r.model.transaction -> g.r.persistence.transaction (#450)
* Refactor g.r.model.transaction -> g.r.persistence.transaction

The ultimate goal here is to eventually delete the model package entirely once
we're no longer using Datastore. To that end, the Cloud SQL stuff that we'll be
keeping needs to be in a different package. Here's the rough outline of the
final desired state I'm thinking of:

g.r.persistence
  .transaction -- what's being moved in this commit
  .converters -- to be moved in another commit
  [maybe more?]
g.r.schema
  [same-ish subpackages as model currently contains]

I'm also trying to think if it makes sense to move the DAO classes out of
g.r.schema, and haven't come up with a good answer yet.

* Also move test resource files

* Merge branch 'master' into mv-tx-to-persistence
2020-01-22 17:19:29 -05:00
Weimin Yu
f2512bc6cc
Check for post-deployment Flyway script changes (#448)
* Check for post-deployment Flyway script changes

Add a script that checks for changes to scripts that
have been deployed to sandbox. This is test in to
be invoked in presubmit and ci.

Extracted common shell functions to an 'rc' file.

Also renamed existing script to be consistent with
other shell scripts.
2020-01-21 15:20:37 -05:00
sarahcaseybot
0b717d40ff
Add dual write for Cursors (#414)
* Add dual write for Cursors

* Fix UpdateCursorCommand to dual write multiple cursors

* Small fixes

* Make UpdateCursorsCommand implement CommandWithCloudSql
2020-01-21 15:04:14 -05:00
Shicong Huang
ad2cf933c2
Add dual read for ReservedList (#423)
* Add dual read for ReservedList

* Extract loading cloud sql list to a method
2020-01-17 15:49:58 -05:00
Weimin Yu
3a9e5d398e
Remove MapUserTypeTest from Integration TestSuite (#449)
* Remove MapUserTypeTest from Integration TestSuite
2020-01-17 14:14:14 -05:00
Michael Muller
2009540f8b
Implement a hibernate type for enums (#446)
* Implement a hibernate type for enums

Leverage GenericCollectionUserType to store enum fields as an array of
strings.

* Changes requested in review

* Remove extraneous import

Curiously, this causes checkstyle to fail now that the import was /removed
from a comment./
2020-01-16 14:56:41 -05:00
sarahcaseybot
ca0d95b066
Fix exDate calculation in domain transfer flows (#389)
* Fix exDate calculation in domain transfer flows

* Link ICANN rule for zero-period transfer

* Get rid of some unnecessary method parameters

* Add domain info commands to test case
2020-01-15 16:43:39 -05:00
Shicong Huang
851465a89d
Relax reserved list existence check to allow Cloud SQL migration (#429) 2020-01-15 13:10:50 -05:00
Shicong Huang
6aa7c19344
Add MapUserType to support converstion between Map and hstore (#443) 2020-01-15 10:37:14 -05:00
Lai Jiang
cf0e8e1b26
Use monitoring client v1.0.7 (#445)
This version of the monitoring client no longer depends on backported
JUnit classes.
2020-01-14 23:11:54 -05:00
Weimin Yu
c90d5ed43b
Fix flaky SQL integration tests (#441)
* Fix flaky SQL integration tests

As a sanity check, the text fixture verifies that all test jdbc connections
have been closed. However, the query to get the connection count is too
general, and may count background admin tasks such as autovacuum and
replication etc, that may appear at any time.
2020-01-14 10:59:55 -05:00
Weimin Yu
e106e25800
Upgradle JUnit to 4.13 (#442)
* Upgradle JUnit to 4.13

Removed third_party/junit folder and all usage of the
JunitBackPort class. As a result, third_party is no
longer a Gradle subproject.

Minor code changes were needed to work around an
error-prone pattern: multiple statement in assertThrows'
runnable lambda.

Also third_party/activation and third_party/jsch. These
dependencies are loaded from remote maven repo. The local
copies are not in use.
2020-01-14 10:54:09 -05:00
Michael Muller
54471e2714
Fix procedure for schema generation (#436)
* Fix procedure for schema generation
2020-01-13 07:41:20 -05:00
Weimin Yu
bd5299b2f2
Print more info during compatibility test (#440)
* Print more info during compatibility test

Prints a summary of the environments to be tested and their
corresponding version tags.
2020-01-10 15:52:27 -05:00
Shicong Huang
4eae10800d
Add user types to support Array in Cloud SQL (#439)
* Add user types to support Array in Cloud SQL

* Remove duplicate null check

* Simplify class structure
2020-01-10 14:54:21 -05:00
Weimin Yu
533515817f
Make Jacoco work with multiple test tasks (#437)
* Make Jacoco work with multiple test tasks

By default Jacoco only looks at execution data from
the 'test' task. This is a problem to 'core' which
has multiple test sets.
2020-01-09 16:32:33 -05:00
Weimin Yu
d4b1048363
Upload Cloud Build schema-deploy config to GCS (#435)
* Upload Cloud Build schema-deploy config to GCS

Forgot to upload cloudbuild-schema-deploy.yaml to GCS.
2020-01-09 15:04:10 -05:00
Weimin Yu
2ffcfc4c5e
Fix outdated comment (#433)
* Fix outdated comment

Removed reference to a testcontainer issues that may no longer
exist.
2020-01-08 16:02:04 -05:00
Weimin Yu
b47e152ca5
Add script to run compatibility tests (#431)
A helper script to run server/schema compatibility tests from
a CI platform. It allows the caller to choose a system under
test (sql or nomulus) and to specify the deployed version
by environment (sandbox or production).
2020-01-08 11:34:09 -05:00
Michael Muller
b523b3835f
Implement changes suggested by testcontainer dev (#426)
* Implement changes suggested by testcontainer dev

See https://github.com/google/nomulus/issues/401
Specifically:
- Use getContainerIpAddress() instead of localhost to insulate us from
  off-machine docker usage.
- Remove shutdown hook to close the container, as testcontainers does this for
  us.
2020-01-08 10:41:36 -05:00
Ben McIlwain
eb7d779bc8 Relax premium list existence check to allow Cloud SQL migration (#428)
* Relax premium list existence check to allow Cloud SQL migration

We need to be able to simultaneously update premium lists that already exist in
Datastore and create them in Cloud SQL (because they haven't been migrated over
yet). This temporarily relaxes the existence check for Cloud SQL so that
"updates" will work even when the list doesn't yet exist there.
2020-01-06 18:09:21 -05:00
Lai Jiang
8fb12d21a2
Use the correct certificate provider type (#427)
TESTED: tested in alpha. Previous I only  tested locally and missed this
typo.
2020-01-06 16:19:50 -05:00
sarahcaseybot
5d9bf64b96
Fix IcannReportingUploadAction to upload reports from the previous month (#425)
* Fix IcannReportingUploadAction to upload reports from the previous month

getFileName now sets the file name of the report to upload to use the month before cursor time.

IcannReportingUploadAction no longer uploads the MANIFEST.txt file since it is not required based on (https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-07#page-9) and the previous implementation of this action did not upload it.

Deletes the ICANN_UPLOAD_MANIFEST cursor since it is no loner needed.

* Add the ICANN_UPLOAD_MANIFEST cursor back
2020-01-06 15:59:01 -05:00
Lai Jiang
76603812e3
Create kzip file for Kythe cross-referencing (#424)
This is set up per b/141716384.

TESTED=Tested on alpha and successfully uploaded the merged kzip file to
GCS.
2020-01-03 16:57:39 -05:00
Shicong Huang
a28754dc5d
Add dual read claims list (#413)
* Add dual read claims list

* Improve warning log and use longer duration for cache

* Extract the comparison logic to a method

* Move cache to DAO
2020-01-03 10:59:34 -05:00
Weimin Yu
d6bf6f375e
Fix flakiness in JodaMoneyConverterTest (#421)
* Fix flakiness in JodaMoneyConverterTest

JpdaMoneyConverterTest relies on Hibernate to deploy its schema.
This introduces an extra jdbc connection in the middle of a test
suite, and may break the connection count checks between tests made
by JpaTransactionManagerRule.

This change updated HibernateSchemaExporter to include Hibernate
proprietary mappings in META-INF/orm.xml.

This change also disabled Hibernate schema push for all tests,
and enabled sql statement logging.
2020-01-02 11:51:20 -05:00
Ben McIlwain
8f67c4b9cf
Support premium list updating in Cloud SQL (#422)
* Support premium list updating in Cloud SQL

This also removes the requirement to specify --also_cloud_sql in nomulus premium
list tooling, instead always persisting to Cloud SQL. It removes a non-USD
premium label in the global test premium list (the Cloud SQL schema doesn't
support a mix of currency units in a single premium list). And it adds a method
to PremiumListDao to grab the most recent version of a given list.

* Merge branch 'master' into premium-lists-always-cloud-sql

* Revert test change

* Create new PremiumListUtils class and refactor out existing method

* Fix tests and update an existing premium price
2020-01-02 11:30:58 -05:00
gbrodman
c17a5c489c
Add unlock fields to RegistryLocks (#408)
* Add unlock fields to RegistryLocks

This will make it easier to reason around inter-connected registry lock
objects (like when we add dependent roids). It will make it easier to
answer the question of "Have all locks associated with this host/contact
roid been unlocked?", as well as the question of "Was the last lock
object associated with this domain unlocked?"

* Responses to CR

* Make the DAO API more specific

* whoops, undo rename
2019-12-30 14:34:06 -07:00
Lai Jiang
dd0e3b7c24
Consolidate certificate supplier module (#410)
* Consolidate certificate supplier module

Both the proxy and the proxy needs certificate suppliers. The PR
consolidates the module that providings those bindings to a shared
module and switched the proxy to use that module. The prober currently
uses P12 file to store its certificates. I am debating keeping that
supplier ro converting them to PEM files for simplicity.

* Rename mode enum values to be more descriptive

* Update annotation names to be more descriptive
2019-12-23 13:09:47 -05:00
Weimin Yu
24d671c070
Separate JPA rules for unit and integration tests (#420)
* Separate JPA rules for unit and integration tests

Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.

* Separate JPA rules for unit and integration tests

Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.
2019-12-19 14:49:54 -05:00
Shicong Huang
f76030f7f0
Add initial support to write/update reserved lists to Cloud SQL (#388)
* Add initial support to write reserved list to Cloud SQL

* Add support to update reserved list in Cloud SQL

* Fix wrong check when override is enabled in create command

* Add sql dependent tests to the suite

* Address comment

* Make invocation of super.execute more readable

* Add test to check upper case and non-puny code label

* Move ReservedListDao related classes to schema package

* Simplify a function name
2019-12-19 12:51:48 -05:00
Shicong Huang
f301314d97
Move ClaimsListDao related classes to new package (#416) 2019-12-19 10:50:41 -05:00
gbrodman
25d43511f6
Add BCC capabilities to the Spec11 reports (#418)
* Add BCC capabilities to the Spec11 reports
2019-12-18 17:08:06 -05:00
Lai Jiang
bfd61ef867
Fix another bug in the proxy (#419)
The promise should be set outside the try block because if we want
warning only, we still want the promise to be set even if the
clientCertificate.checkValidity() throws an error.
2019-12-18 16:24:23 -05:00
Weimin Yu
ffe3eb1548
Remove the schema jar url override for tests (#415)
* Remove the schema jar url override for tests

Revert pull request 374: "Allow schema-loading from arbitrary url in tests".

Pull request 403 (f48e3933f5) is a more
general solution.

This reverts commit 68887d427f.
2019-12-17 11:24:20 -05:00
Weimin Yu
6aaf081489
Create a new app to hold GenerateSqlSchemaCommand (#409)
* Create a new app to hold GenerateSqlSchemaCommand

GenerateSqlSchemaCommand starts postgresql using testcontainer.
This makes junit etc a runtime dependency, allowing them to get
into release artifacts.

By moving this command to a separate tool, we can remove junit
etc as compile/runtime dependency.
2019-12-13 16:05:35 -05:00
Weimin Yu
1143b25391
Fix test on use of external schema jar (#412)
* Fix test on use of external schema jar

One test breaks if path to external schema is set and
test is run from the sqlIntegrationTest task.
2019-12-13 11:32:19 -05:00
Weimin Yu
65cf49f204
Fix sql script name conflict (#411)
* Fix sql script name conflict

There are two V11__ files due to concurrent merge. Renamed one
to V12__

Also removed a @NotNull annotation, which is the fist in the code base.
Most of the code base use @Nullable instead. If we do want to use
@NotNull, we may want to use the javax one instead.
2019-12-12 16:16:43 -05:00
Weimin Yu
f48e3933f5
Run cross-release SQL integration tests (#403)
* Run cross-release SQL integration tests

Run SQL integration tests across arbitrary schema and server
releases.

Refer to integration/README.md in this change for more information.

TESTED=Cloud build changes tested with cloud-build-local
       Used the published jars to test sqlIntegration task locally.
2019-12-12 13:47:49 -05:00
Lai Jiang
9853f23d94
Fix null pointer excpetion bug (#407)
The factory method passes a null trustedCertificates instead of an empty
list.
2019-12-12 13:06:43 -05:00
Ben McIlwain
db7fcf6c38
Add Cloud SQL premium list caches and compare prices with Datastore (#376)
* Add Cloud SQL premium list caches and compare prices with Datastore

Nothing will fail if the prices can't be loaded from Cloud SQL, or if the prices
are different. All that happens is that the error is logged. Then, once this is
running in production for awhile, we'll look at the logs and see if there will
be any pricing implications from switching over to the Cloud SQL version of the
premium lists.

* Add setMaxResults(1) per code review

* Add tests and reorder public functions

* Don't statically import caches

* Improve test pass rate

* Merge branch 'master' into dual-read-premium

* Add PremiumEntry mapping

* Allow update

* Revert column order

* Alphabetize PremiumEntry columns

* Don't bother trying to enforce order

* Private constructor
2019-12-11 16:20:19 -05:00
Weimin Yu
3aad8b6aa7
Use TextDiffSubject to compare multi-line text (#406)
* Use TextDiffSubject to compare  multi-line text

It illustrates differences better.

Moved TextDiffSubject.java to the common project for sharing.
2019-12-10 16:44:59 -05:00
Weimin Yu
f7b243e390
Make devProject a project property (#405)
* Make devProject a project property

Properties set in rootProject's ext block are not overridden by
commandline flags.

* Make flyway commands runnable from FOSS repo

rootProject.devProject, defined in projects.gradle, cannot be
overridden by commandline flags. Added a flag check in :db's
project script.
2019-12-10 11:39:56 -05:00
Michael Muller
e8745f7538
Update lockfiles (#402)
* Update lockfiles

Update lockfiles for the latest dependencies and metadata.
2019-12-10 09:28:44 -05:00
sarahcaseybot
2478a4a93b
Add schema and DAO for cursors in cloudsql (#370)
* Add schema for Cursor

* Add CursorDao and CursorDaoTest

* Fix comment on getTld

* Change tld column to scope

* Fix cursorTime to be converted to DateTime internally and other small fixes

* Add a CursorType enum and a createGlobal constructor for Cursor

* Rename flyway file

* Use cursorType from common/Cursor.java and add null checks
2019-12-09 17:47:06 -05:00
Weimin Yu
bba5aff4b6
Verify RegistryTool can instantiate (#400)
* Verify RegistryTool can instantiate

Add a task that instantiates all command classes in RegistryTool
with runtimeClasspath.

Also make sure that runtimeClasspath is a superset of
compileClasspath.
2019-12-06 12:08:16 -05:00
Michael Muller
df7d272157
Adding junit back into the runtime classpath (#399)
* Adding junit back into the runtime classpath

Unfortunately, GenerateSqlSchemaCommand depends on junit via testcontainers.
We should really move GenerateSqlSchemaCommand out of nomulus tool (we only
use it during development) but this gets nomulus tool working for the time
being.

* Removed unnnecessary trace line.

* lockfiles generated after update_dependency.sh
2019-12-05 16:01:05 -05:00