* Refactor DomainBase into DomainContent and create DomainHistory
This is similar to #587 and #634, but for domains.
One caveat is that we refactor some of the Domain* instance methods to
be static so that they can be called either on DomainBase or
DomainContent, returning the appropriate type each time.
Note that we set DomainHistory to use the same revision ID sequence as
HostHistory and ContactHistory.
In addition, we refactor the tests to the History objects a bit to
reduce duplicate code and because we cannot guarantee yet that the
SQL-stored VKeys are symmetrical -- the ofy keys are not persisted at
the moment.
In addition, rename the DomainHost table to the default Domain_nsHosts so that it automatically creates two separate nsHosts tables for us -- one foreign-keyed on the domain repo ID, and one foreign-keyed on the history revision ID
* Use access hackery to allow manual names for nsHosts tables
* Clean up post merge artifacts
* Add unused setters that Hibernate requires
* Fix the tests and semantic merge conflicts
* Change ns_hosts to ns_host everywhere
* Rename ns_host to host_repo_id
* V42 -> V44
* Add use of interval data type
* Add support for Millis
* Use Java-object type
* Change column type for relock_duration
* add years and months
* Add tests for hours, minutes, and seconds
* Add javadoc describing how joda duration is stored
* Add test for lots of days
* Integrate transaction persistence into JpaTM
Store the serialized transaction whenever we commit from the JPA transaction
manager. This change also adds:
- The Transaction table.
- The TransactionEntity which is stored in it.
- Changes to the test infrastructure to register the TransactionEntity for
tests where we don't load the nomulus schema.
- A new configuration variable to allow us to turn the transaction
persistence functionality on and off (default is "off").
* Changes for review.
* Incremented sequence number of flyway file
* Add lastUpdateTime column to epp resources
Property was inadvertently left out.
Renamed getter and setter to match the property name.
Added a test helper to compare EppResources while ignoring
lastUpdateTime, which changes every time an instance is persisted.
* Create ContactHistory class + table
This is similar to #587, but with contacts instead of hosts.
This also includes a couple cleanups for HostHistoryTest and RegistryLockDaoTest, just making code more proper (we shouldn't be referencing constant revision IDs when using a sequence that is used by multiple classes, and RLDT can extend EntityTest)
Note as well that we set ContactHistory to use the same revision ID sequence as HostHistory.
* Move ContactResource -> ContactBase
* Alter ContactBase and ContactResource
* Update to generic Spec11ThreatMatch table
* Fix SQL syntax
* Make changes to the schema and add a test for null and empty threatTypes
* Fix a small typo
* Change the exception thrown with illegal arguments
Change the import for isNullOrEmpty
* Fix import for checkArgument
* Added a threat to test multiple threat types
* Squash everything together
Create SafeBrowsing_Threats table
Create LocalDateConverter and add indexes to SafeBrowsingThreats
Add indexes to SafeBrowsingThreats and make small style changes
Pass in DateTimeFormatter
Delete LocalDateConverterTest.java
Rebase
Make changes to ThreatType comments
Create LocalDateConverterTest
Add review changes
Add SafeBrowsingThreatTest
Rename repoId, refactor LocalDateConverterTest/SafeBrowsingThreatTest, add foreign keys
Change imports
Add foreign keys and rename version number
Add new generated db-schema file
Clean up null test cases
Add changes
Add foreign keys into SafeBrowsingThreatTeat and apply style checks
Add SafeBrowsingThreatTest into SqlIntegrationTestSuite and change golden file
Make small changes to SafeBrowsingThreatTest
Add tests for ForeignKeyViolations and remove setId in SafeBrowsingThreat
* Change V35 -> V36
* Add a foreign key test for a reference to Registrar
* Move some variables around
* Remove 'fullyQualified' from host and domain names
We don't actually enforce that these are properly fully-qualified
(there's no dot at the end) and we specifically use the term "label
name" when talking about labels.
Note: this doesn't convert FQDN -> DN (et al) in at least two types of
cases:
1. When the term is part of the XML schema
2. When the term is used by some external system, e.g. SafeBrowsing API
* Add TODO to rename fields
* Add proof of concept for HostBase and HostHistory classes
* Use a PROPERTY accessor for @Ids
* Add an unused setter method for Hibernate's sake
* Refactor HostHistory
* Some responses to CR
* Fix relationship and test
* Manually manage the foreign keys for HostHistory
* Protect HostBase's builder and use text for the enum type
* Add responses to CR
- Add javadocs
- Create an ID sequence for host history objects
* Don't try to set the ID
* Use a Long and remove the setter
* Add some comments and rename a couple fields
* Don't change Datastore schema
* Use Long in the Datastore schema
* Add new createVKey method
* Add comments and rename fields
* Rename v27->v31 and regenerate the golden
* Fix superordinateDomain and inetAddresses in HostHistory
* V31 -> V32
* Fix SQL files that got messed up in the merge
* Configure and use a manually-created history ID sequence
* Add three more indices to HostHistory
* Add DomainTransferData and ContactTransferData
* Refactor TransferData to remove unused fields in Contact table
* Add scope for TransferData's type parameter
We'll eventually want to shift everything over to using registrar_id and
registrarId rather than client_id and clientId but for the sake of the
Datastore schema and existing code, we won't change the Java identifier
for now. Once we're completely and only on SQL, we can rename the Java
field easily.
* Create a converter for sets of inetAddresses and use it in HostResource
This can just be a set of strings where each string represents an
address; there's no need for it to be a separate table. This allows
for simplification of the SQL schema.
* Regenerate golden SQL file after renaming v28 -> v29
* Add more tests and rename a typo in the file
* Refactor common test code and use tm methods
* Use JUnit5 API
* Rename test entity
* Add columns for TransferData in Domain and Contact
* Rename flyway file and foreign key
* Rebase on master and address comment
* Compileable commit
* Fix unit test
* Refactor TransferServerApproveEntity
* Use tm().delete(vkeys)
* Rename transfer_period fields
* Rename client_id to registrar_id
* Rebase on master
* Resolve comment
* Rebase on master
* Store the superordinateDomain reference as a VKey rather than Key
This is a reference to a Domain object, so we should store it as a VKey
in reference to the Domain table. This should not affect any business
logic, but rather will allow us to set up the SQL tables for
HostResource et al. properly.
* Generate sql schema for BillingEvent
* Change to use sequence
* Address comments
* Resolve warnings and remove duplicate cost related fields
* Increase the flayway file version to V25
* Remove extra space
* Split to 3 tables, merge VKey
* Rename talbes
* Rename repoId to domainRepoId
* Exclude VKey in schema.txt
* Rename target_id to domain_name
* Fix javadoc
* Resolve comments
* Make VKey persist to datastore as a key
Convert nsHosts entirely to VKey as a proof-of-concept.
Tested as follows:
1) Deployed to crash, verified that nameservers were visible for several
domains (indicating that we are able to load a set of Keys as VKeys)
2) Updated the set of nameservers for a domain (removing some initial
hosts) and verified that the changes went through.
3) Deployed the old version to crash, verified that I was able to retrieve
the newly saved VKeys as Keys.
4) Modified the hosts for the same domain (adding back one of the hosts)
and verified that the change took effect.
5) Redeployed this change to crash, again updated the nameservers to add
another host.
6) Again restored the old version, verified that the new hosts were
visible.
* Changes in response to review
* Convert to a single VKeyTranslatorFactory instance
* Moved vkey field rename to V25
* Convert DomainBase's contacts to VKeys
Convert usage of DomainBase contacts from Key to VKey. This is the same
change as done for nameserver hosts, as it affects all external interfaces.
As with nameserver hosts, we preserve the existing representation so as not to
afffect the datastore representation.
* Add JPA annotations to ContactResource and generate schema
* Resolve comments
* Resolve comments
* Manually add foreign key constraints
* Run with junit5
* Rebase on HEAD
* Fix DomainBaseSqlTest
* Add a registryLockEmailAddress field to RegistrarConctact objects
Because we need to manage the login email, it should be on an account
that we manage. However, for registry lock, we would want to send the
verification emails to a separate email address that the user can use.
As a result, we will use a second field for a user-accessible registry
lock email address. This must be set on the contact when enabling
registry lock for this contact.
* Responses to CR
* derp
* Add a relockDuration to the RegistryLock SQL object
This is the length of time after an unlock that we will re-lock the
domain in question.
* Sort by domain name for stability
Note: this is likely not the best solution for the UI but we can iterate
on this.
* Add nullable
* Add a converter for Duration
* Add a RelockAction and reference to relocks in RegistryLocks
* Respond to CR
- refactor the request param exception logging a bit
- don't log an error if the domain was already locked, just skip
* Save a relock for all locks (if possible)
* derp
* Long -> long + remove unnecessary transact
* semantic merge conflict woo
* fix another semantic merge conflict
* Add the schema and DAO for Locks in CloudSQL
* Addresses some comments
* Change number on flyway file
* Small changes
* More small changes
* Use checkArgumentNotNull instead of checkNotNull
* Address comments
* fix javadocs
* update persistence
* Add JPA annotations to class Registrar
* Use array for Java list
* Exclude parent field
* Use 3 columns for address and use text for enum
* Use EnumParameter and 3 properties in Address
* Rename columns and rebase on HEAD
* Get persistence of DomainBase actually working
Fix all of the existing problems with DomainBase persistence:
- Remove "final" keywords on getters that cause errors during startup.
- Remove Transient from creationTime (since there's a converter for
CreateAutoTimestamp)
- Fix DesignatedContext persistence so that it only creates a single table.
This is a lot more efficient given that these are many-to-one with their
domains.
- Add a flyway script, update the golden schema.
- Create a unit test, add it to the integration test suite.
* Changes request in review
* Regenerated generated schema file.
* Changes for review
* Persist status value enum set
* Changes in response to review
* Changes requested in review
* Fixes for #456
* Rename Domain "status" column to "statuses"
* Add unlock fields to RegistryLocks
This will make it easier to reason around inter-connected registry lock
objects (like when we add dependent roids). It will make it easier to
answer the question of "Have all locks associated with this host/contact
roid been unlocked?", as well as the question of "Was the last lock
object associated with this domain unlocked?"
* Responses to CR
* Make the DAO API more specific
* whoops, undo rename
* Add Cloud SQL premium list caches and compare prices with Datastore
Nothing will fail if the prices can't be loaded from Cloud SQL, or if the prices
are different. All that happens is that the error is logged. Then, once this is
running in production for awhile, we'll look at the logs and see if there will
be any pricing implications from switching over to the Cloud SQL version of the
premium lists.
* Add setMaxResults(1) per code review
* Add tests and reorder public functions
* Don't statically import caches
* Improve test pass rate
* Merge branch 'master' into dual-read-premium
* Add PremiumEntry mapping
* Allow update
* Revert column order
* Alphabetize PremiumEntry columns
* Don't bother trying to enforce order
* Private constructor
* Add schema for Cursor
* Add CursorDao and CursorDaoTest
* Fix comment on getTld
* Change tld column to scope
* Fix cursorTime to be converted to DateTime internally and other small fixes
* Add a CursorType enum and a createGlobal constructor for Cursor
* Rename flyway file
* Use cursorType from common/Cursor.java and add null checks
* Add a converter for CurrencyUnits stored in the database
This uses the well-known String representation for currency units. It also
provides a base class for other converters that will be persisting the
toString() representation.
* Add DB and formatting changes
* Add tests, make minor fixes
* Add Bloom filters to the Cloud SQL PremiumList schema
They are slightly different from the existing Bloom filters stored in Datastore
in that they now use an ASCII String encoding rather than the more generic
CharSequence, and there is no maximum size (whereas we previously had to live
within the 1 MB max entity size for Datastore).
* Add initial support for persisting premium lists to Cloud SQL
This adds support to the `nomulus create_premium_list` command only; support for
`nomulus update_premium_list` will be in a subsequent PR.
The design goals for this PR were:
1. Do not change the existing codepaths for premium lists at all, especially not
on the read path.
2. Write premium lists to Cloud SQL only if requested (i.e. not by default), and
write to Datastore first so as to not be blocked by errors with Cloud SQL.
3. Reuse existing codepaths to the maximum possible extent (e.g. don't yet
re-implement premium list parsing; take advantage of the existing logic), but
also ...
4. Some duplication is OK, since the existing Datastore path will be deleted
once this migration is complete, leaving only the codepaths for Cloud SQL.
* Refactor out common logic
* Add DAO test
* Add tests for parsing premium lists
* Use containsExactly
* Code review changes
* Format
* Re-generate schema
* Fix column names
* Make some tests pass
* Add SQL migration scripts
* Fix test errors
* Regenerate schema using tools command
Rerun GenerateSqlSchemaCommand to pick up RegistryLock
and naming strategy change.
Also updated a new license term which seems to just pop up.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Corrected the type of ClaimsEntry's revision_id column.
It should be plain int8, not bigserial.
Make GenerateSqlSchemaCommand use a custom dialect that
converts all varchar type to 'text' and timestamp to
'timestamptz'.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Re-add other schema classes
* Add Cloud SQL schema for premium lists
This won't work quite yet, pending a solution for the type translator issue
(which will be needed for the currency field, and potentially others).
* Generate basic schema for all of DomainBase
Generate a basic schema for DomainBase and everything that is part of it.
This still isn't complete, in particular it lacks:
- Correct conversions for problematic types (e.g. DateTime, Key...)
- Schema generation for history records.
- Name translation.
