* Restore the original expiration time on domain restore
Except if that time is now in the past, then add a year to it.
* Apply auto-formatter changes to fix my local build
* Merge branch 'master' into restore-expiry-date
* Fix reversed comments
* Use the server host when creating the registry lock verification URL
The app doesn't know about any external configuration that may point to
this app, so there's no way of finding out that, for instance,
registry.google points to the app. Thus, we have to use what the user
gives us so that, in our case, the registry-lock verification
emails can point to https://registry.google/registry-lock-verify instead
of https://domain-registry.appspot.com/registry-lock-verify. The former
is used by clients / users to authenticate, and unfortunately
authenticating on registry.google does not give authentication to
domain-registry.apspot.com.
Tested using the RDAP code that uses getServerName() -- in that case, if
you access registry.google/rdap/<>, it uses registry.google in the URLs
but if you use domain-registry.appspot.com/rdap/<>, it uses
domain-registry.appspot.com in the URLs.
Relatedly, frontend_config_prod-appengine.asciiproto in Piper
is what configures registry.google to point to
domain-registry.appspot.com
* Load CommitLog into BEAM pipeline
Created tools that can generate CommitLogs on a test
datastore.
Defined BEAM transforms that load from CommitLog files and
added a few simple tests.
This is work-in-progress. Next step is to build a consistent
backup by merging Datastore exports and CommitLogs.
We'll eventually want to shift everything over to using registrar_id and
registrarId rather than client_id and clientId but for the sake of the
Datastore schema and existing code, we won't change the Java identifier
for now. Once we're completely and only on SQL, we can rename the Java
field easily.
* Create a converter for sets of inetAddresses and use it in HostResource
This can just be a set of strings where each string represents an
address; there's no need for it to be a separate table. This allows
for simplification of the SQL schema.
* Regenerate golden SQL file after renaming v28 -> v29
* Add more tests and rename a typo in the file
* Refactor common test code and use tm methods
* Use JUnit5 API
* Rename test entity
* Add columns for TransferData in Domain and Contact
* Rename flyway file and foreign key
* Rebase on master and address comment
* Compileable commit
* Fix unit test
* Refactor TransferServerApproveEntity
* Use tm().delete(vkeys)
* Rename transfer_period fields
* Rename client_id to registrar_id
* Rebase on master
* Resolve comment
* Rebase on master
* Store the superordinateDomain reference as a VKey rather than Key
This is a reference to a Domain object, so we should store it as a VKey
in reference to the Domain table. This should not affect any business
logic, but rather will allow us to set up the SQL tables for
HostResource et al. properly.
Given that we currently have no way of reconstituting a symmetric key
from the asymmetric key (or at least, we don't have a 100% reliable way
of doing so) it's best to keep keys as asymmetrical, referring to the
correct database. That way, we don't get situations where we cannot
compare equality of two keys due to one being asymmetrical and one being
symmetrical.
* Remove platformType and threatEntryMetaData fields from ThreatMatch
* Run google-java-format on both files
* Add test for removal of unnecessary fields
* Removed unnecessary fields from Spec11PipelineTest.testEndToEndPipeline_generatesExpectedFiles
* Added style check
* Fix typo
* Load raw records from Datastore export
Created a tool that can export from a test instance of Datastore.
Defined Beam pipeline transforms for loading raw records back from
the export.
This is the first part of the effort to create a consistent snapshot
of Datastore. The next step is to load entity records from CommitLog
files.
* Use TransactionManager for hosts and contacts
Replace Ofy calls with TransactionManager for most interactions involving
hosts and contacts. In the course of this, also convert ForeignKeyIndex and
the EppResourceCache.
* Minor formatting fix
* Refactor LevelDbFileBuilder to accept DS Entity
Builder now can directly work with Datastore Entity objects.
No need to wrap data in ComparableEntity.
* Generate sql schema for BillingEvent
* Change to use sequence
* Address comments
* Resolve warnings and remove duplicate cost related fields
* Increase the flayway file version to V25
* Remove extra space
* Split to 3 tables, merge VKey
* Rename talbes
* Rename repoId to domainRepoId
* Exclude VKey in schema.txt
* Rename target_id to domain_name
* Fix javadoc
* Resolve comments
If the import from Datastore to BigQuery fails, there is no point
enqueuing a job to update the snapshot view.
Also when there's an error updating the snapshot view, log it at severe
level. The HTTP exception thrown is logged at info and triggers a retry
implicitly. I'm not sure if we want this behavior though. Do we want to
retry upon snapshot updating failures? Unless the failurs are transient,
retrying doesn't help. In our case the failure (End of time out of range
in Standard SQL) is not transient.
EntityClasses.ALL_CLASSES has all of our registered entities so
we can use it to initialize VKeyTranslatorFactory.classRegistry
to avoid adding them one by one.
Also, this PR changed to use Key.getKind() to get the kind of
the entity to solve the problem that when the entity class
is an inner class, its kind should still be the class name
instead of OuterClass$InnerClass.
* Read LevelDb incrementally
Made LevelDbLogReader an iterator over a LevelDb data stream,
Reducing memory footprint which is important when used in a
Dataflow pipeline.
* Make VKey persist to datastore as a key
Convert nsHosts entirely to VKey as a proof-of-concept.
Tested as follows:
1) Deployed to crash, verified that nameservers were visible for several
domains (indicating that we are able to load a set of Keys as VKeys)
2) Updated the set of nameservers for a domain (removing some initial
hosts) and verified that the changes went through.
3) Deployed the old version to crash, verified that I was able to retrieve
the newly saved VKeys as Keys.
4) Modified the hosts for the same domain (adding back one of the hosts)
and verified that the change took effect.
5) Redeployed this change to crash, again updated the nameservers to add
another host.
6) Again restored the old version, verified that the new hosts were
visible.
* Changes in response to review
* Convert to a single VKeyTranslatorFactory instance
* Moved vkey field rename to V25
Normal HttpException logs at INFO because they usual do not indicate
anything out of the ordinary and is meant to convey to the client that
there is some expected error. However InternalServerErrorException is
something that we do care about being alerted for so we log it at SEVERE.
* Add TimedTldStateTransitionMapConverter
* Move timedTransitions to a base class and add BillingCostTransitionConverter
* Add test of TimedTransitionPropertyConverterBase
* clean up tests
* Switch tests to JUnit 5
* Make JpaUnitTestRule an extension
* Implement all DatastoreTransactionManager methods
In the course of this:
- Make assertDelete() specific to JpaTransactionManager, remove the return
value from delete()
- Converter "in transaction" assertion to IllegalStateException, which is less
JPA specific.
* Upgraded DatastoreTransactionManagerTest to junit5
* Reflect refunded billing events on deletion in expiration time
This doesn't make any change at the time of the domain deletion itself, but it
will matter if the domain is then undeleted, because we need to know what
expiration date to restore, and if there were any renew or autorenew charges
that were refunded by the deletion because they were in a grace period, they
shouldn't be coming back during the restore.
* Add tests for new expiration date behavior
* Add handling of add/renew grace period overlap
* Convert DomainBase's contacts to VKeys
Convert usage of DomainBase contacts from Key to VKey. This is the same
change as done for nameserver hosts, as it affects all external interfaces.
As with nameserver hosts, we preserve the existing representation so as not to
afffect the datastore representation.
On certain machines (one of mine) the outcastTest consistently fails due
to the following error:
java.lang.NoClassDefFoundError: Could not initialize class
google.registry.persistence.transaction.JpaTestRules$JpaIntegrationTestRule
If I reduce maxparallelForks to 3 it consistently passes. This issue was
mentioned here:
https://discuss.gradle.org/t/junit-test-fails-with-noclassdeffounderror-only-when-maxparallelforks-1/6047
But this post was 8 years old and no solution was identified.
Since we rarely (if ever) want to check the result of a single element load,
make TransactionManager.load(VKey) return non-optional, non-nullable and just
throw a NoSuchElementException if the entity is not found.
Also add a maybeLoad() that does return an optional in case we ever want to do
this (exists() should suffice for an existence check).
* Implement DatastoreEntity and SqlEntity on more classes
For classes that aren't going to transition to SQL, they should just
return an empty list of SqlEntities. When reading these in from the
commit log manifests, we just won't persist anything to SQL.
By having all Datastore entity classes implement DatastoreEntity, we can
avoid potential bugs where we forget to transition some entity to SQL,
or we forget to have the capability to read back from the commit logs.
Note: the EntityTest is still @Ignore'd because there are many SQL and
Datastore classes left -- ones that we are still in the process of
converting or adding, or ones that require more complicated transitions.
Note: Locks and Cursors aren't converted (even though we could) because
they're ephemeral
* Responses to CR
Add a @EntityForTest annotation
fix null that snuck in
* Keep the test ignored for now
* Remove email-editing footgun
Email address is used as the primary key so we should be very careful
about changing it. This will have even more importance when this is the
location to which we will be sending registry lock confirmation emails.
Note: we allow addition or removal of contacts through the UI (and don't
want to disable that) and because all edits are performed by saving the
entire list of contacts, we can't explicitly prevent all possible edits
of email address in the backend. So this doesn't technically prevent
anything security-wise, but it makes it much more difficult to
accidentally edit an email when you shouldn't.
* Enforce non-deletion of registry-lock-enabled contacts
* Fix tests
* Specify contact
* Fix flaky tests due to Entity name conflicts
Objectify siliently replaces current registration of a given kind
when another class is registered for this kind. There are
several TestObject classes in the current code base, which by
default are all mapped to the same kind.
Tests have only been flaky because impacted tests need to run
in specific orders for failures to happen. Using multiple executors
in Gradle also reduced the likely hood of errors. To reproduce the
problem run the following tests in order (e.g., by putting them in
a test suite):
1. ExportCommitLogDiffActionTest
2. CreateAutoTimestampTest
3. RestoreCommitLogsActionTest
In this PR, we
- Made sure all entities have unique kinds.
- Made all test entities register with AppEngineRule instead of directly
with ObjectifyService.
- Added code in AppEngineRule to check for re-registrations.
- Added presumit check for forbidden direct registration.
* Allow the `nomulus renew_domain` command to specify the client ID
This means that a superuser can renew a domain and have the associated history
entry, one time billing event, and renewal grace period be recorded against a
specified registrar rather than the owning registrar of the domain. This is
useful to e.g. renew a domain for free by "charging" the renewal to the
registry's fake registrar. Since the grace period is written to the specified
cliend id as well, if the actual registrar deletes the domain, they don't get
back the money that they didn't pay in the first place.
* Don't NPE when nomulus tool is run without a subcommand
This occurred when an environment was specified but without a subcommand. Now,
the list of valid subcommands is outputted instead of seeing a generic NPE.
This also makes some formatting changes in other files that were causing the
incremental format check to fail.
* Try AppEngineRule
* Add SqlEntity and DatastoreEntity interfaces
These will be used when replaying transactions from either the Datastore
commit logs or the SQL Transaction objects.
When Datastore is the primary database, we will read in the
Datastore commit logs, convert each saved entity to however many SQL
entities, then save those SQL entities in SQL.
When SQL is the primary database, we will read in the SQL objects from a
yet-to-be-created SQL table, convert them to however many Datastore
entities, then save those Datastore entities in Datastore.
This PR includes a couple simple examples of how this will work for entities that are
saveable in both SQL and Datastore (the simple case).
* Add 1-1 mapping between entity annotations and interfaces
* Add JPA annotations to ContactResource and generate schema
* Resolve comments
* Resolve comments
* Manually add foreign key constraints
* Run with junit5
* Rebase on HEAD
* Fix DomainBaseSqlTest
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Hack to call setup and teardown in JUnit5 suite
JUnit 5 runner does not support @BeforeAll and @AfterAll declared
in the Suite class (as opposed to the member classes). However,
staying with the JUnit 4 suite runner would prevent any member
classes from migrating to JUnit 5.
We use a hack to invoke suite-level set up and teardown from tests.
This change is safe in that if the JUnit 5 runner implementation changes
behavior, we will only see false alarms.
* Implement VKeyConverter
Implement a SQL converter that can be used for VKey objects.
Caveats:
- This only works with string columns (there's an excellent chance that all of
our VKeys will use SQL string columns).
- Using this dpesn't establish a foreign key constraint between the referenced
type (the "T" in VKey<T>) and the entity itself: this needs to be
defined manually in the schema.
