Commit graph

3180 commits

Author SHA1 Message Date
Ben McIlwain
e95ce30fa6
Upgrade prober & proxy packages from JUnit 4 to JUnit 5 (#734)
* Upgrade prober & proxy packages from JUnit 4 to JUnit 5
2020-08-01 11:34:58 -04:00
Lai Jiang
090c233592
Some more changes to make it possible to use Java 11 to compile (#732)
Add the class paths of the source files generated by annotation processors to
the javadoc task's class path so that it doesn't complain about missing
Dagger classes.

Also remove empty <p> tags in all generated source files, because jaxb
genrerates files in multiple locations.

Lastly, for unkown reasons when the source level is set to > 8, the core
subproject throws a warning about a Gradle internal annotator processor
that only supports up to Java 8 and cause the Java compliation to fail because
we set -Werror on all java compliation tasks. I don't think there is a
strong reason that we set -Werror anyway, so this commit removes it.
2020-07-31 17:21:28 -04:00
Ben McIlwain
16a31e460c
Get rid of all remaining JUnit 4 usages except in prober & proxy (#731)
* Get rid of all remaining JUnit 4 usages except in prober & proxy subprojects

Caveat: Test suites aren't yet implemented in JUnit 5 so we still use the ones
from JUnit 5 in the core subproject.

* Fix some build errors
2020-07-30 20:29:00 -04:00
Lai Jiang
a02b67caf5
Migrate the documentation package to Java 11 (#729)
* Migrate the documentation package to Java 11

The old Doclet API is deprected and removed in Java 12. This commit
changes the documentation package to use the new recommended API.
However it is not a drop-in replacement and there are non-idiomatic
usages all over the place. I think it is eaiser to keep the current code
logic and kind of shoehorn in the new API than starting afresh as the
return on investment of a do-over is not great.

Also note that the docs package is disabled as of this commit because we
are still using Java 8 to compile which lacks the new API. Once we
switch our toolchains to Java 11 (but still compiling Java 8 bytecode)
we can re-enable this package.

TESTED=ran `./gradlew :docs:test` locally with the documentation package
enabled.
2020-07-30 17:12:33 -04:00
Lai Jiang
bf20a8ef96
Remove empty <p> tag from generated javadoc (#730) 2020-07-30 16:31:25 -04:00
Lai Jiang
8750c07fef
Clean up generated sources configuration (#724)
1. The Gradle apt plugin is no longer needed to process annotations.
2. Without the apt plugin, Gralde puts the source files generated by
   annotation processors in build/generated/sources/annotationProcessor.
3. Change the location of custom generated files to be consistent.
4. Fix a javadoc formatting error.
2020-07-30 15:43:42 -04:00
Weimin Yu
7821de67f8
Run InitSqlPipeline (#727)
* Run InitSqlPipeline

Added the main() method to InitSqlPipeline.

Added a Gradle task to run InitSqlPipeline from command line. This
task is meant for testing and experiments.

Corrected the file name prefix of Datastore export files. Should
be 'output-', defined as 'input-'.
2020-07-30 14:28:53 -04:00
Shicong Huang
1022817384
Fix broken RegistryCli (#728) 2020-07-30 14:10:09 -04:00
Ben McIlwain
8c04bf2599
Rename InjectRule and remove some JUnit4-only dependencies (#726)
* Rename InjectRule and remove some deps

* Merge remote-tracking branch 'upstream/master' into final-core-tests
2020-07-29 20:27:47 -04:00
Ben McIlwain
34116e3811
Clarify un-avail reason on allocation-token-reserved domains (#725)
Apparently, in domain check responses, `avail=false, reason=Allocation token
required` was not sufficiently understood by all registrars. This changes it to
`avail=false, reason=Reserved; alloc. token required` to hopefully make it
crystal clear that the domain in question is reserved, i.e. if you were supposed
to be able to register this domain you'd already know it because we'd have
already given you the requisite allocation token.
2020-07-29 17:13:38 -04:00
Lai Jiang
d180ef43ac
Move the documentation package to its own subproject (#722)
This makes it easier to later migrate the package to Java 11. If we move
and migrate in a single PR, because of the portion of the contents that
s changed, git will have trouble recognizing that some files are
renamed *and* modified and treat them as distinct files, making code
review difficult.
2020-07-29 13:41:02 -04:00
Michael Muller
f55270c46f
Integrate transaction persistence into JpaTM (#717)
* Integrate transaction persistence into JpaTM

Store the serialized transaction whenever we commit from the JPA transaction
manager.  This change also adds:

-   The Transaction table.
-   The TransactionEntity which is stored in it.
-   Changes to the test infrastructure to register the TransactionEntity for
    tests where we don't load the nomulus schema.
-   A new configuration variable to allow us to turn the transaction
    persistence functionality on and off (default is "off").

* Changes for review.

* Incremented sequence number of flyway file
2020-07-28 19:23:44 -04:00
Ben McIlwain
d6d9874da1
Upgrade App Engine and webserver tests from JUnit 4 to 5 (#720)
* Upgrade App Engine and webserver tests from JUnit 4 to 5

* Fix most errors

* Merge branch 'master' into junit5ification

* Fix test server by extracting non-test setup/tear-down

* Merge branch 'master' into junit5ification

* Fix backup tests

* Don't createFile(); asCharSink does it

* Increase the timeout for all WebDriver tests to 60s (helps w/ flakiness)
2020-07-28 14:18:16 -04:00
gbrodman
e0d04cec4f
Set up deployment of the Spec11 pipeline with JPA TM (#716)
* Set up deployment of the Spec11 pipeline with JPA TM

* Remove unnecessarily pipeline options setting

* Use enviroment name in BeamJpaModuleTest

* Fix checkstyle error
2020-07-27 21:04:52 -04:00
Michael Muller
0ce431212e
Add the :nom:generate_golden_schema pseudo-task (#718)
Add a "pseudo-task" in nom_build to do the three step process of generating
the golden schema.  In the course of this, add support for pseudo-tasks in
general, improve the database directory readme and make nom_build not call
gradlew if there are no tasks.
2020-07-27 18:33:16 -04:00
gbrodman
32868b3ab8
Run the (Un)lockDomainCommand in an outer JPA txn (#688)
* Run the (Un)lockDomainCommand in an outer JPA txn

There are a couple things going on here in this commit.

First, we add an external JPA transaction in the
LockOrUnlockDomainCommand class. This doesn't appear to do much, but it
avoids a situation similar to deadlock if an error occurs in Datastore
when saving the domain object. Specifically, DomainLockUtils relies on
the fact that any error in Datastore will be re-thrown in the JPA
transaction, meaning that any Datastore error will back out of the SQL
transaction as well. However, this is no longer true if we are already
in a Datastore transaction when calling DomainLockUtils (unless, again,
we are also in a JPA transaction). Basically, we require that the outer
transaction is the JPA one.

Secondly, this just allows for more breakglass operations in the lock or
unlock domain commands -- in a situation where things possibly go
haywire, we should allow admins to make sure with certainty that a
domain is locked or unlocked.

* Add more robustness and tests for admins locking locked domains

* Fix expected exception message in tests
2020-07-27 18:16:24 -04:00
Shicong Huang
0ecc20b48c
Rename a V40 flyway file to V41 to resolve conflict (#719) 2020-07-27 15:16:01 -04:00
Shicong Huang
c65af4b480
Add remaining columns to Domain's SQL schema (#702) 2020-07-27 13:32:39 -04:00
Legina Chen
3a15a8bdc7
Drop foreign key constraint for Registrar table (#715) 2020-07-27 09:05:30 -07:00
Weimin Yu
9806fab880
Use rearranged sql credentials in flyway task (#712)
* Use rearranged sql credentials in flyway task

Let the flyway tasks use the sql credential files set up for BEAM
pipelines.

Credential files have been created for each environment in GCS
at gs://${project}-beam/cloudsql/admin_credential.enc. All
project editors have access to this file, including the Dataflow
control service account.

Alpha and crash use the 'nomulus-tools-key' in their own project to
decrypt the credential file.

Sandbox and production use the 'nomulus-tools-key' in
domain-registry-dev to decrypt the credential file.

Note that this setup is temporary. It will become obsolete once
we migrate to Cloud Secret Manager for secret storage.
2020-07-24 15:32:01 -04:00
Weimin Yu
6591e0672a
End-to-end Datastore to SQL pipeline (#707)
* End-to-end Datastore to SQL pipeline

Defined InitSqlPipeline that performs end-to-end migration from
a Datastore backup to a SQL database.

Also fixed/refined multiple tests related to this migration.
2020-07-24 09:57:43 -04:00
Ben McIlwain
91b7d92cf8 Upgrade TestPipeline extension from JUnit 4 to 5 2020-07-23 21:21:58 -04:00
Ben McIlwain
33910613da Get presubmits passing
This involves Guava -> Java 8 util migrations and fixing the license header.
2020-07-23 21:21:58 -04:00
Ben McIlwain
1fde678250 Copy TestPipeline rule from Apache Beam project into our codebase
This is copied in here with the absolute minimum # of modifications required
(just a rename to JUnit 5 format and some small fixes required to enable
compilation to be successful).

This is in preparation for the next commit where I'll convert this Rule into a
JUnit 5 extension, which is the entire goal here. But I wanted to get the code
from Apache Beam in with the maximum possible fidelity so that my changes will
be in a separate commit and will thus be obvious.

Note that we do unfortunately need to modify/rewrite the Rule itself; merely
wrapping it in some manner isn't possible.
2020-07-23 21:21:58 -04:00
gbrodman
8d56577653
Don't run presubmits over the .git folder (#711) 2020-07-23 18:12:34 -04:00
Ben McIlwain
3891d411de
Upgrade most of remaining tests from JUnit 4 to JUnit 5 (#708) 2020-07-23 15:43:59 -04:00
gbrodman
cadecb15d8
Rename the email field in UI and include rlock email if it exists (#697)
* Rename the email field in UI and include rlock email if it exists

* Change the capitalization of fields and titles and add a description
2020-07-23 14:30:12 -04:00
gbrodman
9b7f6ce500
Fix some SQL credential issues identified when deploying Beam pipelines (#706)
* Fix some SQL credential issues identified when deploying Beam pipelines

There are two issues fixed here.
1. Without calling `FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create()), the Nomulus tool doesn't know how to handle gs:// scheme files. Thus, if you try to deploy (for instance) the Spec11 pipeline using a GCS credential file, it fails.
2. There was a misunderstanding before about what the credential file
actually refers to -- there is a credential file in JSON format that is
used for gcloud authorization, and there is a space-delimited SQL access
info file that has the instance name, username, and password. These are
separate options and should have separate command-line params.

* Actually we don't need this for remote deployment
2020-07-22 16:52:31 -04:00
Ben McIlwain
cd23748fe8
Upgrade rest of tools test classes to JUnit 5 (#705) 2020-07-22 11:09:21 -04:00
Ben McIlwain
cf41f5d354
Upgrade all remaining flows tests to JUnit 5 (#704) 2020-07-21 19:52:33 -04:00
Ben McIlwain
9a5ba249db
Upgrade converters/TMCH/RDAP to JUnit 5 (#703)
Also renames some existing Rules to Extensions (and removes JUnit 4 features
from them entirely if no longer being used).
2020-07-21 18:48:41 -04:00
Shicong Huang
f5186f8476
Merge two PremiumList entities (#690) 2020-07-21 18:18:52 -04:00
Lai Jiang
4e0ca19d2e
Remove IDN elements from BRDA (#670)
Also added unit tests for RdeStagingReducer.
2020-07-21 15:29:32 -04:00
Ben McIlwain
c812807ab3
Upgrade mapreduce and DNS tests from JUnit 4 to JUnit 5 (#701)
* Upgrade mapreduce and DNS tests from JUnit 4 to JUnit 5

* Merge branch 'master' into junit5-batch-and-dns
2020-07-20 21:33:24 -04:00
Ben McIlwain
9edb43f3e4
Upgrade command test classes from JUnit 4 to JUnit 5 (#700)
* Convert first batch of command tests to JUnit 5

* Upgrade rest of command tests to JUnit 5

* Migrate the last few test classes
2020-07-20 20:45:52 -04:00
gbrodman
b721533759
Create an ImmutableObjectSubject for comparing SQL objects (#695)
* Create an ImmutableObjectSubject for comparing SQL objects

Many times, when comparing objects that are loaded in from / saved to
SQL in tests, there are some fields we don't care about. Specifically,
we might not care about the last update time, revision ID, or other
things like that that are autoassigned by the DB. If we use this, we can
ignore those fields while still comparing the other ones.

* Create an ImmutableObject Correspondence for more flexible usage
2020-07-20 13:14:09 -04:00
gbrodman
ce35f6bc93
Include the user's registry lock email in the lock/unlock modal (#696)
* Include the user's registry lock email in the lock/unlock modal
2020-07-20 12:01:34 -04:00
gbrodman
f7a67b7676
Add a 'Host' parameter to the relock action enqueuer (#699)
* Add a 'Host' parameter to the relock action enqueuer

I believe this is why we are seeing 404s currently -- we should be
specifying the backend host as the target like we do for the
resave-entity async action.
2020-07-17 15:35:44 -04:00
gbrodman
4438944900
Validate potentially-invalid domain names when (un)locking domains (#698)
* Validate potentially-invalid domain names when (un)locking domains
2020-07-17 12:05:19 -04:00
Legina Chen
a22998e1bc
Change clientId to registrarId to resolve the bug wrt the mismatch of the variable names (#692) 2020-07-15 11:03:52 -07:00
Weimin Yu
03d02ab299
Fix JpaIntegrationRule in JUnit4 (#687)
* Fix JpaIntegrationRule in JUnit4

Made DatastoreExtension a JUnit4 Rule.

Nomulus model objects need Datastore API when manipulating Ofy keys.
As a result, JpaIntegrationTestRule must be used with AppEngineRule
or DatastoreExtension.

Also fixed WriteToSqlTest, which is the only JUnit4 test that uses
JpaIntegrationTestRule.
2020-07-15 10:42:33 -04:00
Lai Jiang
47f65f70ab
Fix a typo (#689) 2020-07-15 10:39:11 -04:00
Weimin Yu
1aa1f351bf
Run rdeStaging twice daily in Sandbox (#684)
* Run rdeStaging twice daily in Sandbox

This will allow the cursor to catch up to current date if
it somehow falls behind.
2020-07-14 14:54:34 -04:00
Weimin Yu
94c8c6b9f3
Add lastUpdateTime column to epp resources (#683)
* Add lastUpdateTime column to epp resources

Property was inadvertently left out.

Renamed getter and setter to match the property name.

Added a test helper to compare EppResources while ignoring
lastUpdateTime, which changes every time an instance is persisted.
2020-07-14 14:53:05 -04:00
gbrodman
e74a9e6f02
Allow overrides of ContactBase methods (#681)
Hibernate might (will?) need to override these, so they shouldn't be
final.
2020-07-14 14:47:47 -04:00
gbrodman
37d3cc44b4
Fix small naming issue in a test (#685) 2020-07-14 13:57:44 -04:00
Lai Jiang
c844c8e9b1
Add the ability to parse PKCS#8 private key in PEM file (#682) 2020-07-14 11:20:00 -04:00
gbrodman
f747610533
Include the relock action in the web.xml routing file (#680) 2020-07-13 21:57:35 -04:00
Shicong Huang
e1db357fc3
Merge two reserved list entities (#616)
* Merge reserved list

* Replace INSTANCE with getInstance()

* Fix broken test

* Rebase on master

* Simplify class
2020-07-13 13:40:34 -04:00
Weimin Yu
ba1915e271
Write one PCollection to SQL (#664)
* Write one PCollection to SQL

Defined a transform that writes a PCollection of entities to SQL using
JPA. Allows configuring parallelism level and batch size.
2020-07-13 13:34:01 -04:00