* Add daily cron job for IcannReportingUploadAction
This job checks each ICANN cursor's cursorTime. If all cursorTime values are in the future, the action exits and does nothing. For each cursor that has a cursorTime in the past, the corresponding report is uploaded if it is staged, or logs an error message if the report has not been staged yet.
* Change cron job description
* Add Registry Lock UI
* Responses to CRs, mostly
TODO:
- Figure out wording for the 'not enabled yet' message
- Include the server status change cost in the email, or in the UI?
- Should we show non-completed lock requests in the UI?
* Fix get action test
* Change the not-allowed-for-registrar msg to include support email
* Change the wording on the price
* Move TLD input into the modal, and other changes
- don't log the password
- test to make sure the password shows bullets
* Responses to CR and cleanup
* Format closer to something proper
* Use -t instead of main params for TLDs in nomulus count_domains command
This makes the command consistent with list_domains. I use both frequently and it
was annoying forgetting which one takes -t and which uses main parameters. Now
they both work the same way.
* Add a scrap command to backfill registry locks
* fix tests
* Change comments and messages
* Use URS time (best effort) if one exists
* Don't bother with root cause
* Lazily load jpaTm
This fixes an exception we were seeing in production where commands that only
implemented CommandWithRemoteApi (and that had nothing to do with Cloud SQL)
were nevertheless trying to initialize a JPA TM instance, and then failing.
That stacktrace looked like:
com.google.api.client.http.HttpResponseException: 400 Bad Request
{
"error": "invalid_grant",
"error_description": "Bad Request"
}
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:423)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
at google.registry.keyring.kms.KmsConnectionImpl.attemptDecrypt(KmsConnectionImpl.java:163)
at google.registry.keyring.kms.KmsConnectionImpl.lambda$decrypt$0(KmsConnectionImpl.java:148)
at google.registry.util.Retrier.callWithRetry(Retrier.java:153)
at google.registry.util.Retrier.callWithRetry(Retrier.java:130)
at google.registry.util.Retrier.callWithRetry(Retrier.java:95)
at google.registry.keyring.kms.KmsConnectionImpl.decrypt(KmsConnectionImpl.java:147)
at google.registry.keyring.kms.KmsKeyring.getDecryptedData(KmsKeyring.java:209)
at google.registry.keyring.kms.KmsKeyring.getString(KmsKeyring.java:178)
at google.registry.keyring.kms.KmsKeyring.getToolsCloudSqlPassword(KmsKeyring.java:100)
at google.registry.persistence.PersistenceModule.providesNomulusToolJpaTm(PersistenceModule.java:124)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.proxyProvidesNomulusToolJpaTm(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:61)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:39)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:12)
at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
at google.registry.persistence.DaggerPersistenceComponent.nomulusToolJpaTransactionManager(DaggerPersistenceComponent.java:168)
at google.registry.persistence.transaction.TransactionManagerFactory.createJpaTransactionManager(TransactionManagerFactory.java:38)
at google.registry.persistence.transaction.TransactionManagerFactory.<clinit>(TransactionManagerFactory.java:29)
at google.registry.model.registry.Registries.lambda$createFreshCache$2(Registries.java:60)
at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:243)
at google.registry.model.registry.Registries.getTlds(Registries.java:85)
at google.registry.model.registry.Registries.assertTldsExist(Registries.java:112)
at google.registry.tools.CountDomainsCommand.run(CountDomainsCommand.java:41)
at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:243)
at google.registry.tools.RegistryCli.run(RegistryCli.java:168)
at google.registry.tools.RegistryTool.main(RegistryTool.java:127)
The TL;DR is that RegistryCli was over-eagerly creating the jpaTm, because
there's no reason the Registries cache (which is Datastore-only) should ever
need it, but because this cache is using the Datastore transaction manager, the
other one was being created too.
* Start using JUnit 5
This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration
In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.
* Delete no longer needed lockfiles
* Merge branch 'master' into first-junit5
* Add dual write for Registrar
* Use @AlsoLoad to set street field for Cloud SQL
* Change email body to use the new streetLine field
* Refactored the logic to handle street fields
* Simplify postLoad function
* Address comments
* Add a TODO to remove street
* Add test for onLoad and postLoad
* Rebase on master
* Add the schema and DAO for Locks in CloudSQL
* Addresses some comments
* Change number on flyway file
* Small changes
* More small changes
* Use checkArgumentNotNull instead of checkNotNull
* Address comments
* fix javadocs
* update persistence
* Use JSON API for Maven Repo on GCS
The url pattern https://storage.googleapis.com/{Bucket}/{Path}
uses the legacy XML API, which seems to be less robust than
the JSON API. We have observed connection resets after a few
thousand-file download bursts over 30 minutes.
This PR changes all urls to registry's Maven repo on GCS to
gcs://{Bucket}/{Path}. Gradle uses the JSON API for such urls.
TESTED=In Cloud Build with local change
* Use dependency cache in all Gradle tasks in GCB
Make the initial test and the final publishing steps use the shared
dependency cache.
Also make the initial test use the registry's own maven repo instead
of Maven Central.
* Make Gradle dependency cache shareable in GCB
Make Gradle put its caches in the source tree so that
they can be preserved across steps. When left at their
default location, caches are lost after each step.
* Use CA cert in tests
Our tests were using a self-signed cert created by a netty utility.
That cert is not CA, and cannot be used with newer JDK versions
(starting from u242 with Ubuntu openjdk).
Now we generate a unsafe cert for tests. The new cert,
SelfSignedCaCertificate, needs to be made test-only.
To do so we need to refactor both prober and proxy.
* Always save reserved lists to Cloud SQL
Otherwise it's too easy to forget to run this additional option, and then things will
fall out of sync and start throwing errors.
* Use randomly-generated strings for RegistryLock verification codes
We were using UUIDs before which are also fine, but unnecessarily long.
The RegistryLock class itself does not enforce any particular format for
the lock verification codes.
* Implement dump_golden_schema command in devtool
Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.
Refactor the SQL container functionality from GenerateSqlCommand. There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.
* Reformatted and changes in response to review
* Fix getDockerTag() usage
* Fix "leaked resource"
* Add a RegistryLockPostAction and tests
* Response to CR and more robust email content test
* Fix typo
* Run save + email in a JPA transaction
* Use action in subject
* Allow non-CA cert as trust anchor in tests
Stay with the legacy behavior to unblock release build.
We will update the test cert generation code later.
TESTED=reproduced and fixed test failures in builder container
* Work around Spinnaker issue wrt variables
Cloud Build variable reference need to stay from the ${var} pattern
to prevent Spinnaker from trying to resolve it. In all files that
are used by Spinnaker, we change variable reference to the $var form.
We made the minimum amount of change possible, and will review this
issue after the permanent solution is available.
* Add RegistryLockVerifyAction
The action takes two parameters
- isLock is a boolean, determining whether we're trying to lock or
unlock a domain
- lockVerificationCode is the UUID by which we'll look up the lock
object in question.
The lock in question must not be expired and must be in a valid lockable
/ unlockable state
* Some responses to CR
* Add slash and move test method
* Add more data and tests
* Fix screenshot
* Add JPA annotations to class Registrar
* Use array for Java list
* Exclude parent field
* Use 3 columns for address and use text for enum
* Use EnumParameter and 3 properties in Address
* Rename columns and rebase on HEAD
* Get persistence of DomainBase actually working
Fix all of the existing problems with DomainBase persistence:
- Remove "final" keywords on getters that cause errors during startup.
- Remove Transient from creationTime (since there's a converter for
CreateAutoTimestamp)
- Fix DesignatedContext persistence so that it only creates a single table.
This is a lot more efficient given that these are many-to-one with their
domains.
- Add a flyway script, update the golden schema.
- Create a unit test, add it to the integration test suite.
* Changes request in review
* Regenerated generated schema file.
* Changes for review
* Persist status value enum set
* Changes in response to review
* Changes requested in review
* Fixes for #456
* Rename Domain "status" column to "statuses"
* Require tests on all declared JPA entities
The JPA entity classes declared in persistence.xml reflects
JPA's view of the database. The view must be compatible, though
not always consistent with the actual schema. By defining and
using a new JpaIntegrationWithCoverageRule, this change verifies
that all JPA entities in persistence.xml are included in some
tests in the SqlIntegrationTestSuite.
Also removed one test from SqlIntegrationTestSuite since it
does not use Cloud SQL.
Lastly, global resources such as clock should be provided to
a test fixture. Having one fixture creating its own makes it
harder to use in a multi-system integration test. This change
refactored JpaTransactionManagerRule to take a user provided
clock.
* Change reserved messages for domains w/ required allocation token
In domain check reasons for unavailable domains, we were previously saying that
domains were "Reserved" regardless of whether the domain was fully blocked or
reserved for an anchor tenant or other specific use (e.g. the .new Limited
Release Program). This commit changes the message for the latter situations to
be "Allocation token required", so that registrars have a hint that domain will
show as available if the correct allocation token is supplied to the domain
check command using the allocation token extension.
* Allow claims keys on domain creates during quiet periods
Since we're using TldState.QUIET_PERIOD for the .new LRP (in which only reserved
domains with allocation tokens may be registered), we also need to support
claims keys during this phase. Otherwise, domains that appear in MarksDB will
not be registrable during LRP, even if the correct claims key is provided.
This is based on an error report from a registrar, in which a correct-looking
domain create was failing with the error message "Declared launch extension
phase does not match the current registry phase".
* Add test for checking reserved domain with token
* Allow domain checks with a valid token if the domain isn't fully blocked
* Check specifically for specific-use or anchor reservations
* Add test for LRP in-time token
* Refactor lock/unlock commands to use Registry Locks
* CR responses
* Remove unnecessary XML test files
* Add tests
* Respond to CR
* Refactor further the creation and verification of locks
* remove isUnlocked
* Responses to CR
* Fix tests
* Add admin-override back to unlocking
* Add TODO
* Fix imports
* Refactor g.r.model.transaction -> g.r.persistence.transaction
The ultimate goal here is to eventually delete the model package entirely once
we're no longer using Datastore. To that end, the Cloud SQL stuff that we'll be
keeping needs to be in a different package. Here's the rough outline of the
final desired state I'm thinking of:
g.r.persistence
.transaction -- what's being moved in this commit
.converters -- to be moved in another commit
[maybe more?]
g.r.schema
[same-ish subpackages as model currently contains]
I'm also trying to think if it makes sense to move the DAO classes out of
g.r.schema, and haven't come up with a good answer yet.
* Also move test resource files
* Merge branch 'master' into mv-tx-to-persistence
* Check for post-deployment Flyway script changes
Add a script that checks for changes to scripts that
have been deployed to sandbox. This is test in to
be invoked in presubmit and ci.
Extracted common shell functions to an 'rc' file.
Also renamed existing script to be consistent with
other shell scripts.
* Implement a hibernate type for enums
Leverage GenericCollectionUserType to store enum fields as an array of
strings.
* Changes requested in review
* Remove extraneous import
Curiously, this causes checkstyle to fail now that the import was /removed
from a comment./
* Fix exDate calculation in domain transfer flows
* Link ICANN rule for zero-period transfer
* Get rid of some unnecessary method parameters
* Add domain info commands to test case
