* Update SqlIntegrationTestSuite
Edited Javadoc to emphasize that suite members should be DAO tests.
Removed functional tests from the suite. They do not benefit much
from running against different schemas when the entities they use
are already covered by DAO tests.
Added DomainBaseSqlTest to the suite, which tests DomainBase.
* Create a nom_build wrapper script
nom_build is a wrapper around ./gradlew. It's purpose is to help us
deal with properties. The main problem that it is trying to solve is
that when properties are specified using -P, we don't get an error if
the property we specify isn't correct. As a result, a user or a build
agent can launch a build with unintended parameters.
nom_build consolidates all of the properties that we define into a
python script where the properties are translated to flags (actual
gradlew flags are also proxied). It also generates the property file
and warns the user if the current properties file is out of sync with
the script and includes documentation on each of the properties.
* Enable Cloud SQL when Datastore is enabled for unit test
* Add explanation for why add a ETA field in GenerateEscrowDepositCommand
* Fix line length
* Ignore membershipt test but bring back test suite
* Fix tiny issue
* Listen to the user hitting enter in the lock/unlock modal input fields
Listen to both, just in case one or the other is disabled
* Don't require that the element exist
* Fix broken builds when Maven Central is used
Gradle 6.2.1 apparently introduces a behavior change wrt boolean
expression: empty string used to eval to false, but now evals to
true.
Pre Gradle 6.2.1, root project's Gradle properties apparently were
not set to buildSrc. Now they are passed on to buildSrc -- mavenUrl
in buildSrc changes from null to "".
Both changes break the project when mavenUrl and/or pluginsUrl are
not set on command line.
Also added junit.jupiter-api as testCompile dependencies to projects.
This is a directly used dependency, whose absence causes a Lint
warning.
* Add BasicDao
* Refactor RegistrarDao to extend BasicDao
* Introduce VKey and rewrite BasicDao
* Move CRUD methods to TransactionManager
* Refactor code to simplify the way to get id from entity and sqlKey
* Assert in transaction
* Fix broken test
* Change methods name
* Always validate domain name on allocation token
This is in response to a client-reported error, where they accidentally sent the
wrong domain name on a domain create that included an allocation token. What
should have happened (and that now happens as of this commit) is an error being
thrown that the allocation token does not match the domain name being created.
What happened instead was that, since the incorrectly submitted domain name was
not reserved, the create succeeded (as it would for all creates of unreserved
domains in GA) and the allocation token was redeemed, which is not what you'd
expect.
* Fix tests to reflect changed check behavior
* Refactor DomainLockUtils methods to take a time rather than a clock
* Add administratively (un)lock methods
* Responses to CR
- Javadoc changes
- Method renames
- Variable renames
* Refactor lock methods to use JPA transaction time
* Remove clock, use Datastore transaction time
* Properly use Datastore transaction time, batched
* Continue to throw exceptions on invalid domains
* DAO writes should be in a transaction
* Assume in-transaction for all RLDao methods
* clean up test
* Fix more tests
* add comment
* Add daily cron job for IcannReportingUploadAction
This job checks each ICANN cursor's cursorTime. If all cursorTime values are in the future, the action exits and does nothing. For each cursor that has a cursorTime in the past, the corresponding report is uploaded if it is staged, or logs an error message if the report has not been staged yet.
* Change cron job description
* Add Registry Lock UI
* Responses to CRs, mostly
TODO:
- Figure out wording for the 'not enabled yet' message
- Include the server status change cost in the email, or in the UI?
- Should we show non-completed lock requests in the UI?
* Fix get action test
* Change the not-allowed-for-registrar msg to include support email
* Change the wording on the price
* Move TLD input into the modal, and other changes
- don't log the password
- test to make sure the password shows bullets
* Responses to CR and cleanup
* Format closer to something proper
* Use -t instead of main params for TLDs in nomulus count_domains command
This makes the command consistent with list_domains. I use both frequently and it
was annoying forgetting which one takes -t and which uses main parameters. Now
they both work the same way.
* Add a scrap command to backfill registry locks
* fix tests
* Change comments and messages
* Use URS time (best effort) if one exists
* Don't bother with root cause
* Lazily load jpaTm
This fixes an exception we were seeing in production where commands that only
implemented CommandWithRemoteApi (and that had nothing to do with Cloud SQL)
were nevertheless trying to initialize a JPA TM instance, and then failing.
That stacktrace looked like:
com.google.api.client.http.HttpResponseException: 400 Bad Request
{
"error": "invalid_grant",
"error_description": "Bad Request"
}
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:423)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
at google.registry.keyring.kms.KmsConnectionImpl.attemptDecrypt(KmsConnectionImpl.java:163)
at google.registry.keyring.kms.KmsConnectionImpl.lambda$decrypt$0(KmsConnectionImpl.java:148)
at google.registry.util.Retrier.callWithRetry(Retrier.java:153)
at google.registry.util.Retrier.callWithRetry(Retrier.java:130)
at google.registry.util.Retrier.callWithRetry(Retrier.java:95)
at google.registry.keyring.kms.KmsConnectionImpl.decrypt(KmsConnectionImpl.java:147)
at google.registry.keyring.kms.KmsKeyring.getDecryptedData(KmsKeyring.java:209)
at google.registry.keyring.kms.KmsKeyring.getString(KmsKeyring.java:178)
at google.registry.keyring.kms.KmsKeyring.getToolsCloudSqlPassword(KmsKeyring.java:100)
at google.registry.persistence.PersistenceModule.providesNomulusToolJpaTm(PersistenceModule.java:124)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.proxyProvidesNomulusToolJpaTm(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:61)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:39)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:12)
at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
at google.registry.persistence.DaggerPersistenceComponent.nomulusToolJpaTransactionManager(DaggerPersistenceComponent.java:168)
at google.registry.persistence.transaction.TransactionManagerFactory.createJpaTransactionManager(TransactionManagerFactory.java:38)
at google.registry.persistence.transaction.TransactionManagerFactory.<clinit>(TransactionManagerFactory.java:29)
at google.registry.model.registry.Registries.lambda$createFreshCache$2(Registries.java:60)
at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:243)
at google.registry.model.registry.Registries.getTlds(Registries.java:85)
at google.registry.model.registry.Registries.assertTldsExist(Registries.java:112)
at google.registry.tools.CountDomainsCommand.run(CountDomainsCommand.java:41)
at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:243)
at google.registry.tools.RegistryCli.run(RegistryCli.java:168)
at google.registry.tools.RegistryTool.main(RegistryTool.java:127)
The TL;DR is that RegistryCli was over-eagerly creating the jpaTm, because
there's no reason the Registries cache (which is Datastore-only) should ever
need it, but because this cache is using the Datastore transaction manager, the
other one was being created too.
* Start using JUnit 5
This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration
In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.
* Delete no longer needed lockfiles
* Merge branch 'master' into first-junit5
* Add dual write for Registrar
* Use @AlsoLoad to set street field for Cloud SQL
* Change email body to use the new streetLine field
* Refactored the logic to handle street fields
* Simplify postLoad function
* Address comments
* Add a TODO to remove street
* Add test for onLoad and postLoad
* Rebase on master
* Add the schema and DAO for Locks in CloudSQL
* Addresses some comments
* Change number on flyway file
* Small changes
* More small changes
* Use checkArgumentNotNull instead of checkNotNull
* Address comments
* fix javadocs
* update persistence
* Use JSON API for Maven Repo on GCS
The url pattern https://storage.googleapis.com/{Bucket}/{Path}
uses the legacy XML API, which seems to be less robust than
the JSON API. We have observed connection resets after a few
thousand-file download bursts over 30 minutes.
This PR changes all urls to registry's Maven repo on GCS to
gcs://{Bucket}/{Path}. Gradle uses the JSON API for such urls.
TESTED=In Cloud Build with local change
* Use dependency cache in all Gradle tasks in GCB
Make the initial test and the final publishing steps use the shared
dependency cache.
Also make the initial test use the registry's own maven repo instead
of Maven Central.
* Make Gradle dependency cache shareable in GCB
Make Gradle put its caches in the source tree so that
they can be preserved across steps. When left at their
default location, caches are lost after each step.
* Use CA cert in tests
Our tests were using a self-signed cert created by a netty utility.
That cert is not CA, and cannot be used with newer JDK versions
(starting from u242 with Ubuntu openjdk).
Now we generate a unsafe cert for tests. The new cert,
SelfSignedCaCertificate, needs to be made test-only.
To do so we need to refactor both prober and proxy.
* Always save reserved lists to Cloud SQL
Otherwise it's too easy to forget to run this additional option, and then things will
fall out of sync and start throwing errors.
* Use randomly-generated strings for RegistryLock verification codes
We were using UUIDs before which are also fine, but unnecessarily long.
The RegistryLock class itself does not enforce any particular format for
the lock verification codes.
* Implement dump_golden_schema command in devtool
Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.
Refactor the SQL container functionality from GenerateSqlCommand. There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.
* Reformatted and changes in response to review
* Fix getDockerTag() usage
* Fix "leaked resource"
* Add a RegistryLockPostAction and tests
* Response to CR and more robust email content test
* Fix typo
* Run save + email in a JPA transaction
* Use action in subject
* Allow non-CA cert as trust anchor in tests
Stay with the legacy behavior to unblock release build.
We will update the test cert generation code later.
TESTED=reproduced and fixed test failures in builder container
* Work around Spinnaker issue wrt variables
Cloud Build variable reference need to stay from the ${var} pattern
to prevent Spinnaker from trying to resolve it. In all files that
are used by Spinnaker, we change variable reference to the $var form.
We made the minimum amount of change possible, and will review this
issue after the permanent solution is available.
* Add RegistryLockVerifyAction
The action takes two parameters
- isLock is a boolean, determining whether we're trying to lock or
unlock a domain
- lockVerificationCode is the UUID by which we'll look up the lock
object in question.
The lock in question must not be expired and must be in a valid lockable
/ unlockable state
* Some responses to CR
* Add slash and move test method
* Add more data and tests
* Fix screenshot
* Add JPA annotations to class Registrar
* Use array for Java list
* Exclude parent field
* Use 3 columns for address and use text for enum
* Use EnumParameter and 3 properties in Address
* Rename columns and rebase on HEAD
* Get persistence of DomainBase actually working
Fix all of the existing problems with DomainBase persistence:
- Remove "final" keywords on getters that cause errors during startup.
- Remove Transient from creationTime (since there's a converter for
CreateAutoTimestamp)
- Fix DesignatedContext persistence so that it only creates a single table.
This is a lot more efficient given that these are many-to-one with their
domains.
- Add a flyway script, update the golden schema.
- Create a unit test, add it to the integration test suite.
* Changes request in review
* Regenerated generated schema file.
* Changes for review
* Persist status value enum set
* Changes in response to review
* Changes requested in review
* Fixes for #456
* Rename Domain "status" column to "statuses"
