* Make nomulus compatible with Java 11
This fixes the double-spacing bug with logged EPP XML on App Engine that started
appearing after App Engine switching from using Java 8 to Java 11. Java 9 made
some changes to XML Transformer classes that needed a little bit of work to
accommodate.
This also fixes the unit tests that were failing in Java 11 (all of which were
related to said XML Transformer changes).
* Make code review changes
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.
ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.
Added a mockito dependency, and upgraded mockito-core to 3.3.3.
Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
The added property will by default be null for existing entities and false for new entities. We could write a temporary MapReduce to set it to false for all existing ones, but it seems easier to just modify the query to account for the nullness of the column. The need to check for null won't exist once we migrate the Registry entity to Cloud SQL.
TESTED=deployed to alpha, set the property for tld zombo to true and triggered an export, verified that the column is null for all rows other than the row for zombo.
* Delay SQL availability check during Tool start
During Cloud SQL migration, Cloud SQL password setup in Registry
server requires a few tool commands. Such commands must be able
to run before Cloud SQL is configured.
In this change, we deplay SQL credential loading until access is
needed.
Hibernate complained that ContactResource.getTransferData should not be
final:
WARN: HHH000305: Could not create proxy factory
for:google.registry.model.contact.ContactResource
org.hibernate.HibernateException: Getter methods of lazy classes cannot
be final: google.registry.model.contact.ContactResource#getTransferData
* Set up JpaTransactionManager in BEAM pipelines
Added modules and utilities to create JpaTransactionManager in BEAM
pipelines.
Not wanting to set up AppEngine Remote API to access Keyring in the
Datastore, we instead use the credential files in GCS, which are
used by Spinnaker/Cloud Build and desktop access. Added utility
to download, decrypt, and parse the file. Also added/modified dagger
modules.
* Don't expect a renewal fee on restores when one isn't due
This is a fix on top of #632 so that domain restore commands don't require
acking an illusory renewal fee for 1 year when that isn't actually happening
(i.e. if the domain isn't yet past its original expiration).
Unfortunately, there's still a problem remaining wherein the restore fee on a
domain check will always include the additional year even if it's not
necessary. We don't have a good solution to that. Also note that in versions of
the fee extension more recent than 0.6, the fee extension cannot be passed on a
domain info command at all, so the domain check command is the only way you have
to determine what the restore fee should be. So we definitely do want to get
that right so that the apparent restore fee on a check is the same as the actual
restore fee when running the restore command. We're not quite there yet though
and it's hard to say how we will get there, since we don't load domains during a
domain check command for performance reasons yet we would need to do so in order
to know the expiration date and thus whether the additional year of renewal
should be charged.
A problem for a future PR.
* Squash everything together
Create SafeBrowsing_Threats table
Create LocalDateConverter and add indexes to SafeBrowsingThreats
Add indexes to SafeBrowsingThreats and make small style changes
Pass in DateTimeFormatter
Delete LocalDateConverterTest.java
Rebase
Make changes to ThreatType comments
Create LocalDateConverterTest
Add review changes
Add SafeBrowsingThreatTest
Rename repoId, refactor LocalDateConverterTest/SafeBrowsingThreatTest, add foreign keys
Change imports
Add foreign keys and rename version number
Add new generated db-schema file
Clean up null test cases
Add changes
Add foreign keys into SafeBrowsingThreatTeat and apply style checks
Add SafeBrowsingThreatTest into SqlIntegrationTestSuite and change golden file
Make small changes to SafeBrowsingThreatTest
Add tests for ForeignKeyViolations and remove setId in SafeBrowsingThreat
* Change V35 -> V36
* Add a foreign key test for a reference to Registrar
* Move some variables around
* Add testcontainers' Junit5 support dependency
Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.
Migrated PersistenceModuleTest as an example.
Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
* Fix flakiness caused by ofy entity name conflict
Overrode the 'kind' of two test entities to prevent name conflicts.
Tests are flaky because of this.
Added a check in AppEngineRuleTest for conflicting kinds.
* Show price of reserved domains when using matching allocation token
When the registrar passes the fee extension, this shows the price of the domain
on a check command for reserved domains if the provided allocation token is a
match. Of course, the price is already always displayed on non-reserved names
(regardless of whether the specific provided token is a match or not).
This affects domain checks only; the price is already always displayed on domain
creates because you already by definition have access to register the domain in
question.
* Refactor pipline for Datastore backup loading
Refactored pipeline transforms.
Added testing utilities that handles assertions better.
Investigated and documented challenges in serializing Ofy entities
without side effects.
* Remove 'fullyQualified' from host and domain names
We don't actually enforce that these are properly fully-qualified
(there's no dot at the end) and we specifically use the term "label
name" when talking about labels.
Note: this doesn't convert FQDN -> DN (et al) in at least two types of
cases:
1. When the term is part of the XML schema
2. When the term is used by some external system, e.g. SafeBrowsing API
* Add TODO to rename fields
* Implement a persistable Transaction object
Implement Transaction, which encapsulates a sequence of datastore mutations
that can be serialized and written to the Cloud SQL Transaction table and
subsequently replayed to Datastore from a backend cron job.
* Changes requested in review
* Add a mujtation count to the persisted format
* Fix cookie processing for RDAP URL update
The existing code only does cookie processing on the _first_ Set-Cookie
header. Therefore, if the "id" cookie used for authentication is defined in
anything other than the first Set-Cookie header (as it now is), we don't find
it.
Replace the cookie processing stanza with a line that processes all cookies in
all Set-Cookie headers.
* Add proof of concept for HostBase and HostHistory classes
* Use a PROPERTY accessor for @Ids
* Add an unused setter method for Hibernate's sake
* Refactor HostHistory
* Some responses to CR
* Fix relationship and test
* Manually manage the foreign keys for HostHistory
* Protect HostBase's builder and use text for the enum type
* Add responses to CR
- Add javadocs
- Create an ID sequence for host history objects
* Don't try to set the ID
* Use a Long and remove the setter
* Add some comments and rename a couple fields
* Don't change Datastore schema
* Use Long in the Datastore schema
* Add new createVKey method
* Add comments and rename fields
* Rename v27->v31 and regenerate the golden
* Fix superordinateDomain and inetAddresses in HostHistory
* V31 -> V32
* Fix SQL files that got messed up in the merge
* Configure and use a manually-created history ID sequence
* Add three more indices to HostHistory
* Add DomainTransferData and ContactTransferData
* Refactor TransferData to remove unused fields in Contact table
* Add scope for TransferData's type parameter
* Allow class-specific creation of symmetrict VKeys
When translating from a datastore Key to a VKey, see if the "kind" class
contains a createVKey(com.googlecode.objectify.Key) static method and if it
does, use it to construct a symmetric VKey instead of simply creating an
objectify-sided asymmetric VKey.
As a test case for this, implement the createVKey() static function for
DomainBase. Also, create unit tests for VKeyTranslatorFactory, which
continues to house the functionality.
* Refactor Fee handling so that each fee knows if it's premium
This is a noop for now, as the new isPremium boolean isn't yet used by anything,
but it will be used in follow-up PRs to add additional fee information using the
fee extension (see: b/157621273). Specifically what we're trying to do here is
return <fee:command name="create" standard="1"> (using the finalized version of
the fee extension) when an entire command has no premium fee associated with
it. And in the current earlier versions of the fee extension that we support,
we'll want to display the correct fee amount and class for creates/checks on
reserved domains when a valid allocation token is passed. This also needs the
isPremium information.
There are no testing implications yet because isPremium isn't exposed anywhere,
but there will definitely be lots of test changes once it's feeding into EPP-
visible changes.
* Rename things, add method Javadoc
* Apply formatting
* Restore the original expiration time on domain restore
Except if that time is now in the past, then add a year to it.
* Apply auto-formatter changes to fix my local build
* Merge branch 'master' into restore-expiry-date
* Fix reversed comments
* Use the server host when creating the registry lock verification URL
The app doesn't know about any external configuration that may point to
this app, so there's no way of finding out that, for instance,
registry.google points to the app. Thus, we have to use what the user
gives us so that, in our case, the registry-lock verification
emails can point to https://registry.google/registry-lock-verify instead
of https://domain-registry.appspot.com/registry-lock-verify. The former
is used by clients / users to authenticate, and unfortunately
authenticating on registry.google does not give authentication to
domain-registry.apspot.com.
Tested using the RDAP code that uses getServerName() -- in that case, if
you access registry.google/rdap/<>, it uses registry.google in the URLs
but if you use domain-registry.appspot.com/rdap/<>, it uses
domain-registry.appspot.com in the URLs.
Relatedly, frontend_config_prod-appengine.asciiproto in Piper
is what configures registry.google to point to
domain-registry.appspot.com
* Log client certificate type and length
It appears that most client certificates are RSA certs, but we should
make sure that is indeed the case. Print out the strength of the cert
if it is RSA.
Also adds supports for TLS 1.3 and print out the supported cipher suites.
* Add a comment about zero length certificate
* Make length of non-RSA keys -1
* Don't use TLS 1.3 if JDK SSL provider is used
* Load CommitLog into BEAM pipeline
Created tools that can generate CommitLogs on a test
datastore.
Defined BEAM transforms that load from CommitLog files and
added a few simple tests.
This is work-in-progress. Next step is to build a consistent
backup by merging Datastore exports and CommitLogs.
We'll eventually want to shift everything over to using registrar_id and
registrarId rather than client_id and clientId but for the sake of the
Datastore schema and existing code, we won't change the Java identifier
for now. Once we're completely and only on SQL, we can rename the Java
field easily.
* Create a converter for sets of inetAddresses and use it in HostResource
This can just be a set of strings where each string represents an
address; there's no need for it to be a separate table. This allows
for simplification of the SQL schema.
* Regenerate golden SQL file after renaming v28 -> v29
* Add more tests and rename a typo in the file
* Refactor common test code and use tm methods
* Use JUnit5 API
* Rename test entity
* Add columns for TransferData in Domain and Contact
* Rename flyway file and foreign key
* Rebase on master and address comment
* Compileable commit
* Fix unit test
* Refactor TransferServerApproveEntity
* Use tm().delete(vkeys)
* Rename transfer_period fields
* Rename client_id to registrar_id
* Rebase on master
* Resolve comment
* Rebase on master
* Store the superordinateDomain reference as a VKey rather than Key
This is a reference to a Domain object, so we should store it as a VKey
in reference to the Domain table. This should not affect any business
logic, but rather will allow us to set up the SQL tables for
HostResource et al. properly.
Given that we currently have no way of reconstituting a symmetric key
from the asymmetric key (or at least, we don't have a 100% reliable way
of doing so) it's best to keep keys as asymmetrical, referring to the
correct database. That way, we don't get situations where we cannot
compare equality of two keys due to one being asymmetrical and one being
symmetrical.
* Remove platformType and threatEntryMetaData fields from ThreatMatch
* Run google-java-format on both files
* Add test for removal of unnecessary fields
* Removed unnecessary fields from Spec11PipelineTest.testEndToEndPipeline_generatesExpectedFiles
* Added style check
* Fix typo
* Load raw records from Datastore export
Created a tool that can export from a test instance of Datastore.
Defined Beam pipeline transforms for loading raw records back from
the export.
This is the first part of the effort to create a consistent snapshot
of Datastore. The next step is to load entity records from CommitLog
files.
* Use TransactionManager for hosts and contacts
Replace Ofy calls with TransactionManager for most interactions involving
hosts and contacts. In the course of this, also convert ForeignKeyIndex and
the EppResourceCache.
* Minor formatting fix
* Refactor LevelDbFileBuilder to accept DS Entity
Builder now can directly work with Datastore Entity objects.
No need to wrap data in ComparableEntity.
* Generate sql schema for BillingEvent
* Change to use sequence
* Address comments
* Resolve warnings and remove duplicate cost related fields
* Increase the flayway file version to V25
* Remove extra space
* Split to 3 tables, merge VKey
* Rename talbes
* Rename repoId to domainRepoId
* Exclude VKey in schema.txt
* Rename target_id to domain_name
* Fix javadoc
* Resolve comments
