If the import from Datastore to BigQuery fails, there is no point
enqueuing a job to update the snapshot view.
Also when there's an error updating the snapshot view, log it at severe
level. The HTTP exception thrown is logged at info and triggers a retry
implicitly. I'm not sure if we want this behavior though. Do we want to
retry upon snapshot updating failures? Unless the failurs are transient,
retrying doesn't help. In our case the failure (End of time out of range
in Standard SQL) is not transient.
EntityClasses.ALL_CLASSES has all of our registered entities so
we can use it to initialize VKeyTranslatorFactory.classRegistry
to avoid adding them one by one.
Also, this PR changed to use Key.getKind() to get the kind of
the entity to solve the problem that when the entity class
is an inner class, its kind should still be the class name
instead of OuterClass$InnerClass.
* Read LevelDb incrementally
Made LevelDbLogReader an iterator over a LevelDb data stream,
Reducing memory footprint which is important when used in a
Dataflow pipeline.
* Make VKey persist to datastore as a key
Convert nsHosts entirely to VKey as a proof-of-concept.
Tested as follows:
1) Deployed to crash, verified that nameservers were visible for several
domains (indicating that we are able to load a set of Keys as VKeys)
2) Updated the set of nameservers for a domain (removing some initial
hosts) and verified that the changes went through.
3) Deployed the old version to crash, verified that I was able to retrieve
the newly saved VKeys as Keys.
4) Modified the hosts for the same domain (adding back one of the hosts)
and verified that the change took effect.
5) Redeployed this change to crash, again updated the nameservers to add
another host.
6) Again restored the old version, verified that the new hosts were
visible.
* Changes in response to review
* Convert to a single VKeyTranslatorFactory instance
* Moved vkey field rename to V25
Normal HttpException logs at INFO because they usual do not indicate
anything out of the ordinary and is meant to convey to the client that
there is some expected error. However InternalServerErrorException is
something that we do care about being alerted for so we log it at SEVERE.
* Add TimedTldStateTransitionMapConverter
* Move timedTransitions to a base class and add BillingCostTransitionConverter
* Add test of TimedTransitionPropertyConverterBase
* clean up tests
* Switch tests to JUnit 5
* Make JpaUnitTestRule an extension
* Implement all DatastoreTransactionManager methods
In the course of this:
- Make assertDelete() specific to JpaTransactionManager, remove the return
value from delete()
- Converter "in transaction" assertion to IllegalStateException, which is less
JPA specific.
* Upgraded DatastoreTransactionManagerTest to junit5
* Reflect refunded billing events on deletion in expiration time
This doesn't make any change at the time of the domain deletion itself, but it
will matter if the domain is then undeleted, because we need to know what
expiration date to restore, and if there were any renew or autorenew charges
that were refunded by the deletion because they were in a grace period, they
shouldn't be coming back during the restore.
* Add tests for new expiration date behavior
* Add handling of add/renew grace period overlap
* Convert DomainBase's contacts to VKeys
Convert usage of DomainBase contacts from Key to VKey. This is the same
change as done for nameserver hosts, as it affects all external interfaces.
As with nameserver hosts, we preserve the existing representation so as not to
afffect the datastore representation.
On certain machines (one of mine) the outcastTest consistently fails due
to the following error:
java.lang.NoClassDefFoundError: Could not initialize class
google.registry.persistence.transaction.JpaTestRules$JpaIntegrationTestRule
If I reduce maxparallelForks to 3 it consistently passes. This issue was
mentioned here:
https://discuss.gradle.org/t/junit-test-fails-with-noclassdeffounderror-only-when-maxparallelforks-1/6047
But this post was 8 years old and no solution was identified.
Since we rarely (if ever) want to check the result of a single element load,
make TransactionManager.load(VKey) return non-optional, non-nullable and just
throw a NoSuchElementException if the entity is not found.
Also add a maybeLoad() that does return an optional in case we ever want to do
this (exists() should suffice for an existence check).
* Implement DatastoreEntity and SqlEntity on more classes
For classes that aren't going to transition to SQL, they should just
return an empty list of SqlEntities. When reading these in from the
commit log manifests, we just won't persist anything to SQL.
By having all Datastore entity classes implement DatastoreEntity, we can
avoid potential bugs where we forget to transition some entity to SQL,
or we forget to have the capability to read back from the commit logs.
Note: the EntityTest is still @Ignore'd because there are many SQL and
Datastore classes left -- ones that we are still in the process of
converting or adding, or ones that require more complicated transitions.
Note: Locks and Cursors aren't converted (even though we could) because
they're ephemeral
* Responses to CR
Add a @EntityForTest annotation
fix null that snuck in
* Keep the test ignored for now
* Remove email-editing footgun
Email address is used as the primary key so we should be very careful
about changing it. This will have even more importance when this is the
location to which we will be sending registry lock confirmation emails.
Note: we allow addition or removal of contacts through the UI (and don't
want to disable that) and because all edits are performed by saving the
entire list of contacts, we can't explicitly prevent all possible edits
of email address in the backend. So this doesn't technically prevent
anything security-wise, but it makes it much more difficult to
accidentally edit an email when you shouldn't.
* Enforce non-deletion of registry-lock-enabled contacts
* Fix tests
* Specify contact
* Fix flaky tests due to Entity name conflicts
Objectify siliently replaces current registration of a given kind
when another class is registered for this kind. There are
several TestObject classes in the current code base, which by
default are all mapped to the same kind.
Tests have only been flaky because impacted tests need to run
in specific orders for failures to happen. Using multiple executors
in Gradle also reduced the likely hood of errors. To reproduce the
problem run the following tests in order (e.g., by putting them in
a test suite):
1. ExportCommitLogDiffActionTest
2. CreateAutoTimestampTest
3. RestoreCommitLogsActionTest
In this PR, we
- Made sure all entities have unique kinds.
- Made all test entities register with AppEngineRule instead of directly
with ObjectifyService.
- Added code in AppEngineRule to check for re-registrations.
- Added presumit check for forbidden direct registration.
* Allow the `nomulus renew_domain` command to specify the client ID
This means that a superuser can renew a domain and have the associated history
entry, one time billing event, and renewal grace period be recorded against a
specified registrar rather than the owning registrar of the domain. This is
useful to e.g. renew a domain for free by "charging" the renewal to the
registry's fake registrar. Since the grace period is written to the specified
cliend id as well, if the actual registrar deletes the domain, they don't get
back the money that they didn't pay in the first place.
* Don't NPE when nomulus tool is run without a subcommand
This occurred when an environment was specified but without a subcommand. Now,
the list of valid subcommands is outputted instead of seeing a generic NPE.
This also makes some formatting changes in other files that were causing the
incremental format check to fail.
* Try AppEngineRule
* Remove separate deployment of persistence.xml
We added a step to explicitly copy persistence.xml because for some reason it
wasn't originally getting deployed to app-engine, resulting in failures on
startup. However, this file is now included in core.jar and we are now
getting a warning about multiple persistence units with the same name as it
reads the files from both the filesystem and core.jar.
* Add SqlEntity and DatastoreEntity interfaces
These will be used when replaying transactions from either the Datastore
commit logs or the SQL Transaction objects.
When Datastore is the primary database, we will read in the
Datastore commit logs, convert each saved entity to however many SQL
entities, then save those SQL entities in SQL.
When SQL is the primary database, we will read in the SQL objects from a
yet-to-be-created SQL table, convert them to however many Datastore
entities, then save those Datastore entities in Datastore.
This PR includes a couple simple examples of how this will work for entities that are
saveable in both SQL and Datastore (the simple case).
* Add 1-1 mapping between entity annotations and interfaces
* Add JPA annotations to ContactResource and generate schema
* Resolve comments
* Resolve comments
* Manually add foreign key constraints
* Run with junit5
* Rebase on HEAD
* Fix DomainBaseSqlTest
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Hack to call setup and teardown in JUnit5 suite
JUnit 5 runner does not support @BeforeAll and @AfterAll declared
in the Suite class (as opposed to the member classes). However,
staying with the JUnit 4 suite runner would prevent any member
classes from migrating to JUnit 5.
We use a hack to invoke suite-level set up and teardown from tests.
This change is safe in that if the JUnit 5 runner implementation changes
behavior, we will only see false alarms.
* Implement VKeyConverter
Implement a SQL converter that can be used for VKey objects.
Caveats:
- This only works with string columns (there's an excellent chance that all of
our VKeys will use SQL string columns).
- Using this dpesn't establish a foreign key constraint between the referenced
type (the "T" in VKey<T>) and the entity itself: this needs to be
defined manually in the schema.
* Exclude proxy configs from the FOSS jar
No sensitve data exposed.
Added a todo to modify the release process and stop
building the foss jar on the merged repo.
* Add verification that domain labels aren't multi-level domains
In addition, I did a bit of test refactoring because previously, the
CreateOrUpdateReserveListCommandTestCase test cases weren't actually
testing the proper things -- they were failing with
IllegalArgumentExceptions, but not the right ones.
* Change test name and use IDN library
* Handle numeric labels
String like "0" or "2018" are valid labels but not valid domain names
* Use IDN validation with a dummy TLD
On Arch Linux, DumpGoldenSchemaCommandTest failed due to the follow
error:
java.lang.RuntimeException: Container.ExecResult(exitCode=1, stdout=, stderr=pg_dump: [archiver] could not open output file "/tmp/pg_dump.out": Is a directory)
However I cannot figure out why this permission error happens, as the
docker command is executed as root. Saving the pg_dump output to a
temporary file and copy it over the mapped file works, so I don't
know...
* Migrate SqlIntegrationTestSuite members to Junit5
Made InjectRule and EntityTestCase work with both JUnit4 and 5.
Note that "@RunWith(JUnit4.class)" is no longer needed on
JUnit4 test classes. Therefore, its removal from EntityTestCase
has no impact on child classes. All of them are still included in
tests.
Migrated remaining member classes in SqlIntegrationTestSuite to JUnit5.
* Add Test suite support for JUnit 5 classes
Added Gradle dependencies and updated lockfiles.
Updated SqlInegrationTestSuite to use new annotations.
Migrated one member class in SqlIntegrationTestSuite (CursorDaoTest)
to JUnit 5, and verified that the new Suite runner can handle a
mixture of JUnit 4 and 5 tests in one suite.
Note that Gradle tests that run TestSuites must choose JUnit 4.
Updated core/build.gradle and integration/build.gradle.
* Make AppEngineRule work with JUnit 5
Made AppEngineRule work with both JUnit4 and JUnit 5 and applied
it to PremiumListTest.
Next step is to convert SqlIntegrationTestSuite.java to JUnit5.
* Actually run JUnit 5 tests
In Gradle, JUnit 5 must be explicitly enabled with a call to
test.useJUnitPlatform().
The FilteringTest used in :core must also enable JUnit5 separately.
Fixed AppEngineRule to work with the few tests that have migrated
to JUnit5.
More work is needed with AppEngine before we can migrate tests
that actually use Cloud SQL.
For context, with @EnableRuleMigrationSupport, JUnit 5 runner calls
an external resource's before() and after() methods. TestRule.apply()
is not called, therefore any setup done their will be bypassed with
JUnit 5.
* Upgrade CompareDbBackup for Datastore V3
Upgrade the CompareDbBackup class to work with latest
Datastore backup directory structure.
Also fixed a few unrelated minor issues:
- Remaining cases of improper use of System.setOut
- Wrong import order in one class
