* Add lastUpdateTime column to epp resources
Property was inadvertently left out.
Renamed getter and setter to match the property name.
Added a test helper to compare EppResources while ignoring
lastUpdateTime, which changes every time an instance is persisted.
* Write one PCollection to SQL
Defined a transform that writes a PCollection of entities to SQL using
JPA. Allows configuring parallelism level and batch size.
* Add JUnit Params and start using it
* Convert rest of RDE tests
* Don't check headers for generated tests
* Expand visibility to fix build breakage
* Bump JUnit versions to 5.6.2
* Change Subdomain class to contain domainRepoId
* Remove jpaTm from Spec11PipelineTest and change clientId -> registrarId
* Remove 'client' from a comment
* Include changes to Spec11Pipeline
* add SafeBrowsingTransforms
* Run style
* Use an enum instead of boolean in EntityTestCase constructor
It's more clear to use an enum rather than just a simple boolean
* Add Javadoc and make the enum name more verbose
* Double the # of pubapi instances to better handle traffic spikes
We may also consider switching to an automatic scaling mode soon, on the hope
that it's working better than the last time we tried it (it would help to keep
resource costs down at least).
* Create ContactHistory class + table
This is similar to #587, but with contacts instead of hosts.
This also includes a couple cleanups for HostHistoryTest and RegistryLockDaoTest, just making code more proper (we shouldn't be referencing constant revision IDs when using a sequence that is used by multiple classes, and RLDT can extend EntityTest)
Note as well that we set ContactHistory to use the same revision ID sequence as HostHistory.
* Move ContactResource -> ContactBase
* Alter ContactBase and ContactResource
* Verify that the RegistryLock input has the correct registrar ID
We already verify (correctly) that the user has access to the registrar
they specify, but nowhere did we verify that the registrar ID they used
is actually the current sponsor ID for the domain in question. This is
an oversight caused by the fact that our testing framework only uses
admin accounts, which by the nature of things have access to all
registrars and domains.
In addition, rename "clientId" to "registrarId" in the RLPA object
* Change the wording on the incorrect-registrar message
* Output PO number in detailed report
The PO number header was added during the beam migration but we forgot
to print the actual data in the corresponding column. This resulted in a
misalignment of columns in the detailed report.
This PR fixes it. Note that we cannot drop PO number from the header (as is not
useful in the detailed report) because the header represents all fields
that are to be parsed from the SQL query results, and PO number *is*
needed when generating the invoice itself. By dual-purposing the header
(both as the required fields in the parser and the first line in the
detailed report) we have to include the value of PO number in the
detailed report CSV as well.
* Disambiguate injected Cloud SQL parameter names
This allows us to also inject the BeamJpaModule into RegistryTool, which
allows us to use the SocketJpaTransactionManager in Beam pipelines.
Some side effects of this include:
- duplication of KMS connections -- one standard, one Beam
- duplication of the creation of the partial Hibernate SQL configs
- removal of ambiguity between credentialFileName, credentialFilename,
and credentialFilePath -- we now use the latter.
- Performing the credential null check when instantiating the SQL
connection rather than when instantiating the module object. See the code
comments for more details on this.
I verified that this compiles and the tests run successfully when
injecting a @SocketFactoryJpaTm into a Beam pipeline.
* Remove two unnecessary config points and change the name of two params
* Use @Config instead of @Named and change the pool size
* Replace non-visible link with code
* Load Datastore snapshot from backup files
Defined a composite transform that loads from a Datastore export and
concurrent CommitLog files, identify entities that still exist at the
end of the time window, and resolve their latest states in the window.
* Exclude Test/Monitoring Registrars from escrow
Registrars used for testing and monitoring should not be included
in Data escrow. They also lack the required ianaIdentifier property
and would fail ICANN data validation.
Note that since alpha and crash environments have bad data that
break the RDE process, we need to verify this change in Sandbox.
* Make EppResource.loadCached() use batched fetch
Use a batched fetch (ofy().load().keys(...)) from datastore in
EppResource.loadCached().
To support this, convert TransactionManager.load(Iterable<VKey>) to accept the
more flexible generic parameters and return a map.
* Simplify datastore key streaming
* Changes requested in review.
* Update to generic Spec11ThreatMatch table
* Fix SQL syntax
* Make changes to the schema and add a test for null and empty threatTypes
* Fix a small typo
* Change the exception thrown with illegal arguments
Change the import for isNullOrEmpty
* Fix import for checkArgument
* Added a threat to test multiple threat types
This adds an entriesToImmutableMap() collector that can be used in place of
toImmutableMap(Map.Entry::getkey, Map.Entry::getValue()).
It also fixes up some existing calls that use toImmutableMap() when terser
alternatives exist.
* Make nomulus compatible with Java 11
This fixes the double-spacing bug with logged EPP XML on App Engine that started
appearing after App Engine switching from using Java 8 to Java 11. Java 9 made
some changes to XML Transformer classes that needed a little bit of work to
accommodate.
This also fixes the unit tests that were failing in Java 11 (all of which were
related to said XML Transformer changes).
* Make code review changes
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.
ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.
Added a mockito dependency, and upgraded mockito-core to 3.3.3.
Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
The added property will by default be null for existing entities and false for new entities. We could write a temporary MapReduce to set it to false for all existing ones, but it seems easier to just modify the query to account for the nullness of the column. The need to check for null won't exist once we migrate the Registry entity to Cloud SQL.
TESTED=deployed to alpha, set the property for tld zombo to true and triggered an export, verified that the column is null for all rows other than the row for zombo.
* Delay SQL availability check during Tool start
During Cloud SQL migration, Cloud SQL password setup in Registry
server requires a few tool commands. Such commands must be able
to run before Cloud SQL is configured.
In this change, we deplay SQL credential loading until access is
needed.
Hibernate complained that ContactResource.getTransferData should not be
final:
WARN: HHH000305: Could not create proxy factory
for:google.registry.model.contact.ContactResource
org.hibernate.HibernateException: Getter methods of lazy classes cannot
be final: google.registry.model.contact.ContactResource#getTransferData
* Set up JpaTransactionManager in BEAM pipelines
Added modules and utilities to create JpaTransactionManager in BEAM
pipelines.
Not wanting to set up AppEngine Remote API to access Keyring in the
Datastore, we instead use the credential files in GCS, which are
used by Spinnaker/Cloud Build and desktop access. Added utility
to download, decrypt, and parse the file. Also added/modified dagger
modules.
* Don't expect a renewal fee on restores when one isn't due
This is a fix on top of #632 so that domain restore commands don't require
acking an illusory renewal fee for 1 year when that isn't actually happening
(i.e. if the domain isn't yet past its original expiration).
Unfortunately, there's still a problem remaining wherein the restore fee on a
domain check will always include the additional year even if it's not
necessary. We don't have a good solution to that. Also note that in versions of
the fee extension more recent than 0.6, the fee extension cannot be passed on a
domain info command at all, so the domain check command is the only way you have
to determine what the restore fee should be. So we definitely do want to get
that right so that the apparent restore fee on a check is the same as the actual
restore fee when running the restore command. We're not quite there yet though
and it's hard to say how we will get there, since we don't load domains during a
domain check command for performance reasons yet we would need to do so in order
to know the expiration date and thus whether the additional year of renewal
should be charged.
A problem for a future PR.
* Squash everything together
Create SafeBrowsing_Threats table
Create LocalDateConverter and add indexes to SafeBrowsingThreats
Add indexes to SafeBrowsingThreats and make small style changes
Pass in DateTimeFormatter
Delete LocalDateConverterTest.java
Rebase
Make changes to ThreatType comments
Create LocalDateConverterTest
Add review changes
Add SafeBrowsingThreatTest
Rename repoId, refactor LocalDateConverterTest/SafeBrowsingThreatTest, add foreign keys
Change imports
Add foreign keys and rename version number
Add new generated db-schema file
Clean up null test cases
Add changes
Add foreign keys into SafeBrowsingThreatTeat and apply style checks
Add SafeBrowsingThreatTest into SqlIntegrationTestSuite and change golden file
Make small changes to SafeBrowsingThreatTest
Add tests for ForeignKeyViolations and remove setId in SafeBrowsingThreat
* Change V35 -> V36
* Add a foreign key test for a reference to Registrar
* Move some variables around
* Add testcontainers' Junit5 support dependency
Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.
Migrated PersistenceModuleTest as an example.
Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
* Fix flakiness caused by ofy entity name conflict
Overrode the 'kind' of two test entities to prevent name conflicts.
Tests are flaky because of this.
Added a check in AppEngineRuleTest for conflicting kinds.
* Show price of reserved domains when using matching allocation token
When the registrar passes the fee extension, this shows the price of the domain
on a check command for reserved domains if the provided allocation token is a
match. Of course, the price is already always displayed on non-reserved names
(regardless of whether the specific provided token is a match or not).
This affects domain checks only; the price is already always displayed on domain
creates because you already by definition have access to register the domain in
question.
* Refactor pipline for Datastore backup loading
Refactored pipeline transforms.
Added testing utilities that handles assertions better.
Investigated and documented challenges in serializing Ofy entities
without side effects.
* Remove 'fullyQualified' from host and domain names
We don't actually enforce that these are properly fully-qualified
(there's no dot at the end) and we specifically use the term "label
name" when talking about labels.
Note: this doesn't convert FQDN -> DN (et al) in at least two types of
cases:
1. When the term is part of the XML schema
2. When the term is used by some external system, e.g. SafeBrowsing API
* Add TODO to rename fields
* Implement a persistable Transaction object
Implement Transaction, which encapsulates a sequence of datastore mutations
that can be serialized and written to the Cloud SQL Transaction table and
subsequently replayed to Datastore from a backend cron job.
* Changes requested in review
* Add a mujtation count to the persisted format
* Fix cookie processing for RDAP URL update
The existing code only does cookie processing on the _first_ Set-Cookie
header. Therefore, if the "id" cookie used for authentication is defined in
anything other than the first Set-Cookie header (as it now is), we don't find
it.
Replace the cookie processing stanza with a line that processes all cookies in
all Set-Cookie headers.
* Add proof of concept for HostBase and HostHistory classes
* Use a PROPERTY accessor for @Ids
* Add an unused setter method for Hibernate's sake
* Refactor HostHistory
* Some responses to CR
* Fix relationship and test
* Manually manage the foreign keys for HostHistory
* Protect HostBase's builder and use text for the enum type
* Add responses to CR
- Add javadocs
- Create an ID sequence for host history objects
* Don't try to set the ID
* Use a Long and remove the setter
* Add some comments and rename a couple fields
* Don't change Datastore schema
* Use Long in the Datastore schema
* Add new createVKey method
* Add comments and rename fields
* Rename v27->v31 and regenerate the golden
* Fix superordinateDomain and inetAddresses in HostHistory
* V31 -> V32
* Fix SQL files that got messed up in the merge
* Configure and use a manually-created history ID sequence
* Add three more indices to HostHistory
* Add DomainTransferData and ContactTransferData
* Refactor TransferData to remove unused fields in Contact table
* Add scope for TransferData's type parameter
* Allow class-specific creation of symmetrict VKeys
When translating from a datastore Key to a VKey, see if the "kind" class
contains a createVKey(com.googlecode.objectify.Key) static method and if it
does, use it to construct a symmetric VKey instead of simply creating an
objectify-sided asymmetric VKey.
As a test case for this, implement the createVKey() static function for
DomainBase. Also, create unit tests for VKeyTranslatorFactory, which
continues to house the functionality.
* Refactor Fee handling so that each fee knows if it's premium
This is a noop for now, as the new isPremium boolean isn't yet used by anything,
but it will be used in follow-up PRs to add additional fee information using the
fee extension (see: b/157621273). Specifically what we're trying to do here is
return <fee:command name="create" standard="1"> (using the finalized version of
the fee extension) when an entire command has no premium fee associated with
it. And in the current earlier versions of the fee extension that we support,
we'll want to display the correct fee amount and class for creates/checks on
reserved domains when a valid allocation token is passed. This also needs the
isPremium information.
There are no testing implications yet because isPremium isn't exposed anywhere,
but there will definitely be lots of test changes once it's feeding into EPP-
visible changes.
* Rename things, add method Javadoc
* Apply formatting
