Commit graph

125 commits

Author SHA1 Message Date
Rachel Guan
62e5de8a3a
Add support for delay of duration when scheduling a task (#1493)
* Add support for delay by duration when scheduling task

* Fix comments

* Add test for negative duration

* Change delay parameter type to duration
2022-02-03 22:25:39 -05:00
Rachel Guan
bbe5c058fe
Add support for empty or null params for createTask() (#1448)
* Add support for null or empty params

* Add Null or empty check in CollectionUtils

* Remove content type header for empty params in POST request
2022-01-13 12:44:41 -05:00
Lai Jiang
7e115fa23e
Upgrade to Beam 2.34.0 (#1472)
This version of Beam does not have an explicit dependency on log4j.
There are a couple of other things that need to change due to the
upgrade.

1) The new version pulls in a dependency that is not on Maven Central
but on packages.confluent.io, so we need to explicitly add this repo.

2) The new version has a dependency on flogger 0.6 anb above , which removed
the LoggerConfig class (see google/flogger#142).
We therefore backported the class. In the long term we should do what
was suggested in the issue and use the normal JDK Logger config
directly.

3) The intSqlPipeline dependency graph also needs to be updated.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1472)
<!-- Reviewable:end -->
2021-12-19 22:48:15 -05:00
Rachel Guan
548ae25fac
Change Optional::isEmpty to Optional::isPresent (#1428) 2021-11-18 17:08:15 -05:00
Rachel Guan
d76abfc23a
Change TaskQueueUtils to CloudTaskUtils in CommitLogFanoutAction (#1408)
* Change TaskOptions to Task in CommitLogFanoutAction

* Add a createTask method that takes clock and jitterSeconds

* Change CreateTask parameter type and improve test cases

* Improve comments and test casse

* Improve test cases that handel jitterSeconds
2021-11-17 10:54:42 -05:00
Rachel Guan
2218663d55
Add VKey to String and String to VKey methods (#1396)
* Add stringify and parse methods to SerializeUTils

* Improve comments and test cases

* Fix comments and test strings

* Fix dependency warning
2021-11-02 13:25:35 -04:00
Weimin Yu
93a479837f
Make entities serializable for DB validation (#1401)
* Make entities serializable for DB validation

Make entities that are asynchronously replicated between Datastore and
Cloud SQL serializable so that they may be used in BEAM pipeline based
comparison tool.

Introduced an UnsafeSerializable interface (extending Serializable) and
added to relevant classes. Implementing classes are allowed some
shortcuts as explained in the interface's Javadoc. Post migration we
will decide whether to revert this change or properly implement
serialization.

Verified with production data.
2021-10-28 12:19:09 -04:00
Ben McIlwain
a451524010
Add tests for obscure hostname canonicalization rule (#1388)
Also correctly configures Gradle for the util subproject (it wasn't possible to
run tests in IntelliJ without these changes).
2021-10-14 14:53:28 -04:00
Lai Jiang
62eb8801c5
Finish RDE pipeline implementation in SQL mode (#1330)
This PR adds the final step in RDE pipeline (enqueueing the next action
  to Cloud Tasks) and makes some necessary changes, namely by making all
  CloudTasksUtils related classes serializable, so that they can be used
  on Beam.

<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1330)
<!-- Reviewable:end -->
2021-10-04 21:02:44 -04:00
Ben McIlwain
3caee5fba7
Improve some log messages for readability/consistency (#1333)
* Improve some log messages for readability/consistency

* Address code review comments
2021-09-27 11:35:14 -04:00
Ben McIlwain
8d421e995e
Rename client ID to registrar ID in most places (#1317)
* Rename client ID to registrar ID in most places

This is a code-only change, that shouldn't require any sort of data
migration. Correspondingly, there are some existing uses of clientId that are
not migrated (e.g. Datastore fields, task queue payloads, URL parameters for
actions that might be hit from task queues, etc.). And it of course doesn't
modify any fields in EPP XML. Note that the Cloud SQL schema fields are
already named using the registar_id pattern.

This also doesn't yet touch on the -c parameters in nomulus tools; that will be
coming later (since that is an external manual touch-point, it will require a
lot more in the way of changes to various meta scripts and documentation).

* Change more client IDs

* Merge branch 'master' into clientid-to-registrarid
2021-09-16 12:57:43 -04:00
Lai Jiang
1a26677d72
Implement a util class to manage push queues using Cloud Tasks API (#1290)
* Implement a util class to manage push queues using Cloud Tasks API

Push queues were part of App Engine when they debuted. As a result the
Task Queue API were part of the App Engine SDK and can only be used in
App Engine classic runtime. The new Cloud Tasks API can be used in any
runtime but it only supports push queues. In this PR we implement a util
class (CloudTasksUtils) like TaskQueueUtils to handle enqueuing tasks to
push queues using Cloud Tasks. One action (TldFanoutAction) was
converted to use the new API as a demo. Mass migration of other call sites of
the old API will follow in a separate PR.

TESTED=deployed to alpha and verified that tasks are corrected enqueued
and executed.
2021-08-24 21:13:54 -04:00
Ben McIlwain
5158673f21
Consolidate all Registry/TLD-related classes into google.registry.model.tld (#1277) 2021-08-11 18:04:51 -04:00
Lai Jiang
676616a172
Remove the use of GCS APIs provided from GAE SDK (#1228)
The API provided by the GAE SDK will not be available outside GAE
runtime. This presents a problem when we migrate off of GAE. More
pressingly, the RDE pipeline migration to Beam requires that we write to
GCS on GCE. Previously we were able to sidestep the issue by delegating
the writes to FileIO provided by Beam, which knows how to write to GCS.
However the RDE pipeline cannot use FileIO directly as it needs to write
to multiple files in one go and explicit use of GCS API is needed.

An unfortunate side effect of the API migration is that the new testing
library contains a bug which makes serializing GcsUtils impossible. It
is fixed upstream but not released yet. The fix has been backported for
the time being.

<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1228)
<!-- Reviewable:end -->
2021-07-13 14:52:37 -04:00
Rachel Guan
f5d344d5c9
Add cc support to email service (#1230)
* Add cc support to email service
2021-07-08 12:03:03 -04:00
Ben McIlwain
e5801e1b60
Update more dependencies to newer versions (#999)
* Update more dependencies to newer versions

* Add lockfiles and back out 2 problematic dep updates

* Fix the build (backs out more changes)

* Back out qdox 2.0 too
2021-03-10 11:22:07 -05:00
Ben McIlwain
a52a8695e3
Update a lot of Cloud API dependencies (#994)
Hopefully this will help us to resolve our 'gRPC is too old' warning (gRPC isn't
a direct dependency so it's likely coming in through these).
2021-03-09 13:24:41 -05:00
Lai Jiang
4e44a98139
Update a few plugins for Java 11 compatibility (#966)
* Update a few plugins for Java 11 compatibility

Guice 5.0.1 is now compatible with Java 11. However we don't
directly depend on Guice. Rather Soy depends on Guice. So I added a
direct dependency on Guice 5.0 just before Soy in order to frontload Soy
and pull in the newer version.

Mockito 3.7.7 is now compatible with Java 11. The complication is that
we need to use the inline version of Mockito, which among other things
also allows mocking for final classes (hooray!). It will eventually
become the default Mockito mock maker but for now it needs to be
manually activated.

Note that the inline version now introduces another warning:

```
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
```

Which I think is WAI due to how the inline mock maker works. Waiting on
the author to confirm.

After these to changes the only illegal reflective access is caused by
App Engine SDK tools, which we will rid ourselves of when we migrate off
of GAE.

* Restore package-lock.json
2021-03-02 21:01:30 -05:00
Weimin Yu
ffe3124ee1
Properly set up JPA in BEAM workers (#965)
* Properly set up JPA in BEAM workers

Sets up a singleton JpaTransactionManger on each worker JVM for all
pipeline nodes to share.

Also added/updated relevant dependencies. The BEAM SDK version change
caused the InitSqlPipeline's graph to change.
2021-02-22 18:11:32 -05:00
sarahcaseybot
13f61dd7b9
Add string constants for HTTP header names (#956)
* Add string constants for HTTP header names

* revert package-lock changes

* Clarify names

* add CONTENT_TYPE

* Fix formatting

* Move X-FORWARDED-FOR to ProxyHttpHeaders
2021-02-11 12:02:51 -05:00
Lai Jiang
b59d2ae419
Upgrade to Gradle 6.8.2 (#957) 2021-02-08 09:35:43 -05:00
sarahcaseybot
22d3612be3
Convert Strings to X509 Certificates before validating (#948)
* Convert certificate strings to certificates

* Format fixes

* Revert "Format fixes"

This reverts commit 26f88bd313.

* Revert "Convert certificate strings to certificates"

This reverts commit 6d47ed2861.

* Convert strings to certs for validation

* Add clarification comments

* Add test to verify endoded cert from proxy

* Add some helper methods

* add tests for PEM with metadata

* small changes

* replace .com with .test
2021-01-29 16:59:57 -05:00
Ben McIlwain
fcd79e7c18
Add a full ASCII StringGenerator alphabet (#882)
* Add a full ASCII StringGenerator alphabet

This is intended for the purpose of generating maximally secure passwords for
PostgreSQL and others. We may need to remove a few of these punctuation
characters if they prove to be more trouble than they're worth (e.g. backtick).
2020-11-19 18:08:04 -05:00
Weimin Yu
ae6b414b82
Add a Secret Manager client for Nomulus (#872)
* Add a Secret Manager client for Nomulus
2020-11-12 17:12:52 -05:00
sarahcaseybot
8bd5eb4eca
Move CertificateChecker to core/ (#852)
* Move CertificateChecker to core/

* rename certificates/ to certs/
2020-10-30 15:57:12 -04:00
sarahcaseybot
576c05ff5f
Add certificate checks to RegistrarSettingsAction (#843)
* Add certificate checks to RegistrarSettingsAction

* Add some comments

* Add more functionality to CertificateChecker and update call sites

* Small code cleanups

* Small format fix
2020-10-23 15:46:57 -04:00
Weimin Yu
6ed286e3bc
Upgrade error-prone to 3.3.4 (#848)
* Upgrade error-prone to 3.3.4

This would fix the failure with openjdk 11.0.9 in
3.3.3.

Fixed new antipatterns raised by the new version:
- Replaced unnecessary lambdas with methods.
- Switched wait/sleep calls to equivalent methods using java.time types
- Types inheriting Object.toString() should not be assigned to string
parameter in logging statements.
2020-10-23 11:17:57 -04:00
sarahcaseybot
93d922af6f
Add certificate checks for create and update registrar commands (#837)
* Add certificatechecks for create and update registrar commands

* Add CertificateCheckerModule

* Remove commented out code

* Still tring to get dependency injection to work

* Get this actually working

* Add tests for multiple violations

* Small formatting fixes

* Rename configs and fix collectors

* Add checks for failover client certificate

* Fix formatting
2020-10-22 11:43:22 -04:00
gbrodman
0b73e9032c
Use a SQL date object for LocalDates (#842)
* Use a SQL date object for LocalDates

* Clean up comment
2020-10-20 15:44:23 -04:00
Ben McIlwain
2e230664fd
Convert CertificateViolation into an enum (#829)
* Convert CertificateViolation into an enum

This ends up being nicer to deal with from callsites than class instances, while
still permitting full configurability of all parameters. There are various other
changes/fixes as well.
2020-10-07 22:19:36 -04:00
sarahcaseybot
6ab69d4226
Add a CertificateChecker class (#793)
* CertificateChecker with checks for expiration and key length

* Add validity length check

* Get rid of hard-coded constants and DSA checks

* add files that for some reason weren't included in last commit

* Rename violations and other fixes

* Add displayMessage to CertificateViolation enum

* Switch violations from an enum to a class

* small changes

* Get rid of ECDSA checks

* add checks for old validity length

* Change error message for validity length
2020-10-06 15:47:42 -04:00
gbrodman
1bba68dd96
Add success/failure notifications for the RelockDomainAction (#733)
* Add success/failure notifications for the RelockDomainAction

If a relock fails for some reason, we should noisily notify both our
alerting email and also the registry lock contacts for the registrar in
question. The consequences of a silent failure could be large so it's
something we want to avoid if at all possible.

In addition, we only retry tasks up to two times (one in 5min, one in
10min).

This model of retries / notifications, as well as the language contained
in the emails, have been LGTMed by Bruno and Kirsten

* Change the wording on the success email

* Change the times in which we send emails

For transient failures:
- Retry every ten minutes for six hours
- Send an email after a half hour (three failures) saying that we'll
retry
- Send a success email if we succeed any time after that

For non-transient failures:
Send an email with the error message and don't retry

* Add a test for the max-failure-email

* Responses to CR

- retry indefinitely
- send an email to just the alert address if we can't find the lock
- refactor the task enqueuer a bit

* non-transient -> non-retryable

* Use a lenient stubber for the AESU

* Add a DS transaction around the re-lock
2020-08-31 14:15:47 -04:00
Lai Jiang
a302cf3e04
Update dependencies to get rid of some illegal reflection warnings (#768)
For some inexplicable reasons I have to move the javax.mail package one
spot up to avoid its classes being shadowed by those provided in the
appengine package...
2020-08-14 16:53:22 -04:00
Ben McIlwain
b2a78b5d68
Allow allocation token discounts on premiums and for multiple years (#744)
* Allow allocation token discounts on premiums and for multiple years

* Add domain check flow tests

* Address code review comments

* Update schema file
2020-08-05 17:54:47 -04:00
Lai Jiang
9304e2f421
Enable Java 11 features (#737)
* Enable Java 11 features

As of this commit Java 11 must be used to build. The generated bytecode
is still at Java 8 due to App Engine task queue limit.

Also fixed a bug where the included google-java-format jar file is not
used, requiring the user to install it separately.

See: https://cloud.google.com/appengine/docs/standard/java/taskqueue/push
2020-08-03 16:08:29 -04:00
Ben McIlwain
16a31e460c
Get rid of all remaining JUnit 4 usages except in prober & proxy (#731)
* Get rid of all remaining JUnit 4 usages except in prober & proxy subprojects

Caveat: Test suites aren't yet implemented in JUnit 5 so we still use the ones
from JUnit 5 in the core subproject.

* Fix some build errors
2020-07-30 20:29:00 -04:00
Lai Jiang
8750c07fef
Clean up generated sources configuration (#724)
1. The Gradle apt plugin is no longer needed to process annotations.
2. Without the apt plugin, Gralde puts the source files generated by
   annotation processors in build/generated/sources/annotationProcessor.
3. Change the location of custom generated files to be consistent.
4. Fix a javadoc formatting error.
2020-07-30 15:43:42 -04:00
Ben McIlwain
4cb88ab6e7
Convert RDE tests (and some test rules) from JUnit 4 to JUnit 5 (#677)
* Add JUnit Params and start using it

* Convert rest of RDE tests

* Don't check headers for generated tests

* Expand visibility to fix build breakage

* Bump JUnit versions to 5.6.2
2020-07-10 21:32:36 -04:00
Ben McIlwain
74b2de5c35
Make ImmutableMap Stream collect()ion nicer (#654)
This adds an entriesToImmutableMap() collector that can be used in place of
toImmutableMap(Map.Entry::getkey, Map.Entry::getValue()).

It also fixes up some existing calls that use toImmutableMap() when terser
alternatives exist.
2020-06-26 11:57:26 -04:00
Weimin Yu
2e5466f32f
Fix flaky tests that depends on order of rules
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.

ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.

Added a mockito dependency, and upgraded mockito-core to 3.3.3.

Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
2020-06-24 23:14:47 -04:00
Weimin Yu
2c243a7d5f
Add testcontainers' Junit5 support dependency (#640)
* Add testcontainers' Junit5 support dependency

Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.

Migrated PersistenceModuleTest as an example.

Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
2020-06-22 13:12:58 -04:00
Ben McIlwain
df8ce38796
Rename whitelist -> allow list (#635)
* Rename whitelist -> allow list

* Merge branch 'master' into allowlist-denylist
2020-06-18 18:36:05 -04:00
Lai Jiang
bd443633f6
Add a task to compile javadoc across all packages (#597)
Also fixes various issues that prevent javadoc compliation.
2020-05-27 10:33:46 -04:00
Weimin Yu
1911c11623
Add Test suite support for JUnit 5 classes (#549)
* Add Test suite support for JUnit 5 classes

Added Gradle dependencies and updated lockfiles.

Updated SqlInegrationTestSuite to use new annotations.

Migrated one member class in SqlIntegrationTestSuite (CursorDaoTest)
to JUnit 5, and verified that the new Suite runner can handle a
mixture of JUnit 4 and 5 tests in one suite.

Note that Gradle tests that run TestSuites must choose JUnit 4.
Updated core/build.gradle and integration/build.gradle.
2020-04-07 21:06:49 -04:00
Shicong Huang
db7d49801d
Supress exccesive logging message from Cloud SQL (#540)
* Supress exccesive logging message from Cloud SQL

* Upgrade package versions that were downgraded before
2020-03-31 17:57:18 -04:00
Weimin Yu
6bd50421bc
Fix broken builds when Maven Central is used (#509)
* Fix broken builds when Maven Central is used

Gradle 6.2.1 apparently introduces a behavior change wrt boolean
expression: empty string used to eval to false, but now evals to
true.

Pre Gradle 6.2.1, root project's Gradle properties apparently were
not set to buildSrc. Now they are passed on to buildSrc -- mavenUrl
in buildSrc changes from null to "".

Both changes break the project when mavenUrl and/or pluginsUrl are
not set on command line.

Also added junit.jupiter-api as testCompile dependencies to projects.
This is a directly used dependency, whose absence causes a Lint
warning.
2020-03-10 11:21:03 -04:00
Lai Jiang
08a6a333ad
Upgrade to Gradle 6.2.1 (#501) 2020-03-05 18:47:25 -05:00
Ben McIlwain
bbacdb9704
Start using JUnit 5 (#488)
* Start using JUnit 5

This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration

In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.

* Delete no longer needed lockfiles

* Merge branch 'master' into first-junit5
2020-02-19 18:29:59 -05:00
Weimin Yu
e106e25800
Upgradle JUnit to 4.13 (#442)
* Upgradle JUnit to 4.13

Removed third_party/junit folder and all usage of the
JunitBackPort class. As a result, third_party is no
longer a Gradle subproject.

Minor code changes were needed to work around an
error-prone pattern: multiple statement in assertThrows'
runnable lambda.

Also third_party/activation and third_party/jsch. These
dependencies are loaded from remote maven repo. The local
copies are not in use.
2020-01-14 10:54:09 -05:00
Ben McIlwain
8f67c4b9cf
Support premium list updating in Cloud SQL (#422)
* Support premium list updating in Cloud SQL

This also removes the requirement to specify --also_cloud_sql in nomulus premium
list tooling, instead always persisting to Cloud SQL. It removes a non-USD
premium label in the global test premium list (the Cloud SQL schema doesn't
support a mix of currency units in a single premium list). And it adds a method
to PremiumListDao to grab the most recent version of a given list.

* Merge branch 'master' into premium-lists-always-cloud-sql

* Revert test change

* Create new PremiumListUtils class and refactor out existing method

* Fix tests and update an existing premium price
2020-01-02 11:30:58 -05:00