Commit graph

3557 commits

Author SHA1 Message Date
sarahcaseybot
c729a30d38
Add ECDSA key validation to Certificate Checker (#855)
* Add ecdsa key validation

* Add some comments

* fix merge conflicts

* change variable names

* Separate tests

* separate curve tests
2020-11-09 15:28:48 -05:00
gbrodman
acce1a7d3b
Add DuallyWrittenEntity convenience interface (#866)
* Add DuallyWrittenEntity convenience interface

For classes that are dually written we don't need to worry about replay
conversion, so this just allows us to remove unnecessary unimportant
methods

* Rename DWE to NonReplicatedEntity and enable test
2020-11-09 15:13:40 -05:00
gbrodman
0c6363c04f
Persist two singleton entities in SQL tables (#860)
* Persist two singleton entities in SQL tables

A table might not be the best place to store singleton entities, but by
doing this we ensure we can easily inspect them later and use the same
sort of persistence logic for these that we do elsewhere.

ServerSecret is stored upon retrieval so that we make sure that the same
secret is used in both Datastore and SQL (we wouldn't want to change
it).

* Responses to CR

* Don't have a separate ID for the singleton entities

* Rename secret UUID

* Rename and regenerate
2020-11-09 13:47:42 -05:00
Shicong Huang
cb764b5d30
Convert Registry realted tests to working with SQL (#862) 2020-11-09 12:10:55 -05:00
Shicong Huang
420f3bf380
Add missing foreign key constraints (#845)
* Add missing foreign key constraints

* Fix failed unit tests
2020-11-09 10:55:29 -05:00
Michael Muller
7097b0f5e6
Allow InjectExtension to apply in "before each" (#864)
* Allow InjectExtension to apply in "before each"

InjectExtension is non-standard in that, while it cleans up the changes it has
applied in its "after each" method, those changes must be applied by the test
driver or by another extension.  This breaks extension ordering, which is
something we're going to need to make heavy use of for the ReplayExtension.

Allow changes to be specified (but not activated) after construction and apply
all unactivated changes during "before each."
2020-11-09 07:52:55 -05:00
Shicong Huang
30d57d9476
Upgrade Postgres realted packages to latest version (#861) 2020-11-06 10:14:44 -05:00
sarahcaseybot
700d612ff9
Remove November 1 check (#859) 2020-11-04 13:10:48 -05:00
Shicong Huang
e3d400958c
Add a command to fix duplicate id issue for BillingEvent.Recurring (#798)
* Refactor ResaveEntitiesWithUniqueIdCommand to extract common methods

* Add a command to dedupe BillingEvent.Recurring entities
2020-11-03 16:11:56 -05:00
gbrodman
aa84d5d138
Fix entity annotations for a few classes (#856)
* Fix entity annotations for a few classes

- Abstract classes shouldn't implement DatastoreEntity/SqlEntity
- We aren't persisting Modification in SQL
- Because DelegationSignerData is embedded, we don't need to worry about
converting and persisting it
- DomainDsDataHistory isn't persisted in Datastore
2020-11-03 10:28:34 -05:00
gbrodman
d685f7e2df
Add a SQL schema and DAO for KmsSecretRevision (#840)
* Add a SQL schema and DAO for KmsSecretRevision

The dual-object nature of KmsSecret and KmsSecretRevision will not be
necessary once we have moved to SQL. In that world, the only object will
be the one now called KmsSecretRevision. KmsSecretRevision already
stores its parent so all we need to do is convert that key to the String
secretName (or from the secretName to the key, if loading from SQL) and
select the max revision ID for a given secret name.

In a future PR, we will add a dual-writing DAO to these objects and
perform the dual writes, similar to how ReservedList functions.

* Regenerate diagram

* Rename revisionId and cryptoKeyVersionName

* Fix SQL files and diagram
2020-10-30 18:45:43 -04:00
gbrodman
40eef2a06c
Add SQL schema and DAO for SignedMarkRevocationList (#850)
* Add SQL schema and DAO for SignedMarkRevocationList

This gets saved every day so we're not concerned about history, meaning
we can dual-write and/or dual-read without concern. The structure here
is somewhat similar to the ClaimsListDao and related classes.

* Update the DB files
2020-10-30 17:52:09 -04:00
sarahcaseybot
8bd5eb4eca
Move CertificateChecker to core/ (#852)
* Move CertificateChecker to core/

* rename certificates/ to certs/
2020-10-30 15:57:12 -04:00
Weimin Yu
83918e92b5
Sync the live folder after Nomulus rollback (#854)
* Sync the live folder after Nomulus rollback

To update the nomulus tool on corp desktop, the artifacts from the
rollback target release should be copied to the 'live' folder.

* Fix a test
2020-10-29 16:21:56 -04:00
Shicong Huang
5bba65835a
Change primary key of DelegationSignerData and add its history table (#841)
* Change primary key of DelegationSignerData and add its history table

* Change primary key and resolve comments

* Rebase on HEAD
2020-10-29 16:19:15 -04:00
Shicong Huang
1e51f51979
Use TransactionManager APIs in DatastoreHelper (#849)
* Make DatastoreHelper support Postgresql

* Rebase on HEAD

* Resolve comments

* Use put* inside insert* and update*

* Resolve comments
2020-10-29 11:41:04 -04:00
Weimin Yu
db2e896d42
An automated rollback tool for Nomulus (#847)
* An automated rollback tool for Nomulus

A tool that directs traffic between deployed versions. It handles the
conversion between Nomulus tags and AppEngine versions, executes schema
compatibility tests, ensures that steps are executed in the correct order,
and updates deployment records appropriately.
2020-10-29 10:37:20 -04:00
Michael Muller
478064f32b
Add a test for ImmutableObject significant fields (#853)
This should have been introduced in #846.  Better late than never.
2020-10-28 11:11:42 -04:00
Shicong Huang
0db535b838
Add ER diagram links to db/README.md (#851) 2020-10-28 10:13:39 -04:00
Shicong Huang
3705f37fab
Add a build task to upload ER diagrams to GCS (#844)
* Add a build task to upload ER diagrams to GCS

* Merge ER diagram task into cloudbuild-javadoc
2020-10-27 10:41:12 -04:00
Michael Muller
86bdd154bc
Restore ofy keys in GracePeriod objects (#846)
* Restore ofy keys in GracePeriod objects

Restore the ofy keys when loading GracePeriod object from SQL.  There's no
clear way to do this using the normal approach (fix-up during a PostLoad
method) because fixups to these violate immutability after hibernate has
already obtained their hash values.  Instead, we force reconstitution of the
ofy keys in all public methods that access them (including equals() and
hashCode()) so that they can be generated before an invalid hash is generated.

As part of this change, convert the GracePeriod id from an autogenerated
sequence to a UUID allocated from ObjectifyService and enhance ImmutableObject
to allow it to exclude certain fields from hash/equals and print.

The ImmutableObject enhancements are necessary because we compare grace
periods against locally created test objects in a number of unit tests and
there's no way this can work with GracePeriods loaded from SQL currently, as
they will have an identifier field generated from the database and the test
objects will have an identifier field of null (or a new unique value, after
this change).

Removing autogeneration from GracePeriod ids ended up being likely not
strictly necessary for this change (it was a consequence of an earlier
iteration).  However, it does alleviate the problem of mutation of an
immutable object after creation and is more in line with how we've decided to
allocate other identifiers.

* Changed needed after rebase.
2020-10-26 13:38:14 -04:00
sarahcaseybot
576c05ff5f
Add certificate checks to RegistrarSettingsAction (#843)
* Add certificate checks to RegistrarSettingsAction

* Add some comments

* Add more functionality to CertificateChecker and update call sites

* Small code cleanups

* Small format fix
2020-10-23 15:46:57 -04:00
gbrodman
f52e887db5
Create SQL schema for RdeRevision (#835)
* Create SQL schema for RdeRevision

* Split RdeRevision IDs into three separate DB fields as unified pkey

* Rename variable

* Merge remote-tracking branch 'origin/master' into rdeRevision

* Rename variable in one other location

* Implement no-op toDatastore/Sql for RdeRevision

* Responses to CR

* Merge remote-tracking branch 'origin/master' into rdeRevision

* Use a date for the date column

* Fix exception messages in tests

* Regen diagram to fix the test

* Use assignment in static factory methods

* Merge remote-tracking branch 'origin/master' into rdeRevision
2020-10-23 13:14:07 -04:00
Weimin Yu
6ed286e3bc
Upgrade error-prone to 3.3.4 (#848)
* Upgrade error-prone to 3.3.4

This would fix the failure with openjdk 11.0.9 in
3.3.3.

Fixed new antipatterns raised by the new version:
- Replaced unnecessary lambdas with methods.
- Switched wait/sleep calls to equivalent methods using java.time types
- Types inheriting Object.toString() should not be assigned to string
parameter in logging statements.
2020-10-23 11:17:57 -04:00
sarahcaseybot
93d922af6f
Add certificate checks for create and update registrar commands (#837)
* Add certificatechecks for create and update registrar commands

* Add CertificateCheckerModule

* Remove commented out code

* Still tring to get dependency injection to work

* Get this actually working

* Add tests for multiple violations

* Small formatting fixes

* Rename configs and fix collectors

* Add checks for failover client certificate

* Fix formatting
2020-10-22 11:43:22 -04:00
gbrodman
0b73e9032c
Use a SQL date object for LocalDates (#842)
* Use a SQL date object for LocalDates

* Clean up comment
2020-10-20 15:44:23 -04:00
Shicong Huang
4d5d9700b8
Add a command to generate ER diagram for SQL schema (#839)
* Add a command to generate ER diagram for SQL schema

* Add graphviz as runtime dependency

* Update ER diagrams for #838
2020-10-15 17:31:43 -04:00
Michael Muller
3534a146e4
Restore ofy keys in DomainTransferData (#838)
* Restore ofy keys in DomainTransferData

Restore composite VKeys correctly in DomainTransferData (they were previously
missing their ofy keys).

* Use "AlsoLoad" to populate history ids
2020-10-15 07:54:47 -04:00
gbrodman
4ec7f23e84
Use the parent to store the history repo ID and fill in the base object (#830)
* Use the parent to store the history repo ID and fill in the base object

Storing the repo ID in the parent and in the base object has two primary
benefits.

First, it unifies the parent information in the HistoryEntry's `parent`
object. This simplifies the builders and the data flow.

Second, when possible (which should be always, post-migration) we fill
out the DomainContent's repo ID (similarly for the other EPP resources)
which means that when reconstituting the ofy keys we don't need to pass
the repo ID in from a separate object. This way, all the data are
encapsulated where they should be.

The primary downside here is that it further reduces the "immutability"
of the history objects (since we're using the Hibernate setter for the
parent repo ID) but we weren't immutable anyway.

* Respond to CR

- compare the entire vkeys in tests
- always return the parent for repo ID

* Simplify creation of parent VKeys

* Fix flipped isAssignableFrom check in VKey

* Merge remote-tracking branch 'origin/master' into historyRepoId
2020-10-09 16:01:51 -04:00
Shicong Huang
7a68b1b6f0
Revert package-lock.json to version from #676 (#834)
Co-authored-by: gbrodman <gbrodman@google.com>
2020-10-09 15:57:23 -04:00
Shicong Huang
14e593d9e1
Add SchemaCrawler as dependency (#833) 2020-10-09 15:02:11 -04:00
Weimin Yu
2d5de96fbd
Minor python changes (#832)
* Minor python changes

Use dataclasses instead of attrs. The former is part of the standard lib
while the latter may need to be installed separately.

Also added python3 to the list of prerequisites.
2020-10-09 14:50:21 -04:00
Weimin Yu
13d30b0bfb
Maintain a release-to-Version map in deployment (#831)
* Maintain a release-to-Version map in deployment

Keep track of the mapping between Nomulus release tags and AppEngine
version ids with a mapping file. This is necessary because AppEngine
does not support custom versioning. With this mapping, rollbacks could
be automated. Automation of rollbacks is important since there are
test-supporting metadata to be updated, but are easily forgotten.

During the last stage of deployment, current per-service version ids
are fetched using gcloud and are appended to a file on GCS. Each line
is of the format "{RELEASE_TAG},{APPENGINE_SERVICE},{APPENGINE_VERSION}.

This change has been tested in crash. The rollback script is still a
work in progress.
2020-10-09 13:32:52 -04:00
Shicong Huang
b05f6b4ba3
Add SQL schema for DelegationSignerData (#713)
* Add SQL schema for DelegationSignerData

* Remove join table

* Rebased on HEAD

* Rebase on head
2020-10-09 10:22:31 -04:00
Shicong Huang
17a1387184
Disable auto-generation on id for HostHistory and ContactHistory (#827) 2020-10-08 12:30:54 -04:00
Ben McIlwain
2e230664fd
Convert CertificateViolation into an enum (#829)
* Convert CertificateViolation into an enum

This ends up being nicer to deal with from callsites than class instances, while
still permitting full configurability of all parameters. There are various other
changes/fixes as well.
2020-10-07 22:19:36 -04:00
Michael Muller
299b093f78
Correctly restore composite VKeys in DomainContent (#825)
* Restore composite vkeys in DomainContent

PollMessage/BillingEvent vkeys in DomainContent must have their ofy keys
restored from other fields in DomainContent (namely the repo id and their
specific history event ids).

Add PostLoad methods to DomainContent and DomainHistory to do the restoration.

* Fixes for review.

* Deal with foreign-key cycles
2020-10-07 12:42:01 -04:00
Ben McIlwain
61e7fa89f7
Fix incorrect repackaged App Engine import (#828)
* Fix incorrect repackaged App Engine import
2020-10-07 11:52:17 -04:00
sarahcaseybot
6ab69d4226
Add a CertificateChecker class (#793)
* CertificateChecker with checks for expiration and key length

* Add validity length check

* Get rid of hard-coded constants and DSA checks

* add files that for some reason weren't included in last commit

* Rename violations and other fixes

* Add displayMessage to CertificateViolation enum

* Switch violations from an enum to a class

* small changes

* Get rid of ECDSA checks

* add checks for old validity length

* Change error message for validity length
2020-10-06 15:47:42 -04:00
gbrodman
0f09a4a0ab
Add more Datastore/Sql Entity annotations (#826)
* Add more Datastore/Sql Entity annotations

* Move comments up a line
2020-10-05 13:07:53 -04:00
Shicong Huang
95f6ccc657
Fix vkey reconstruction for PollMessage (#823)
* Fix vkey reconstruction for PollMessage

* Add foreign key

* Rebase on HEAD
2020-10-05 10:35:40 -04:00
Michael Muller
77fabe4dc4
Move "WithLongVKey" to BillingEvent subclasses (#821)
When loading the VKeys for the BillingEvents hierarchy, it is necessary to
restore the original concrete class for the type, otherwise we end up with a
different (and incompatible) VKey.

As part of this, convert the cancellation matching billing event to
VKey<Recurring>, which seems like the only thing it actually can be.
2020-10-02 15:20:23 -04:00
Lai Jiang
71fa12f773
Fix invoicing SQL (#824) 2020-10-01 14:29:49 -04:00
Shicong Huang
fd40a6a2b9
Use composite primary key for HostHistory and ContactHistory (#809)
* Use composite primary key for HostHistory and ContactHistory

* Update flyway file version

* Make getters private

* Add javadoc

* Rebase on HEAD
2020-10-01 11:01:57 -04:00
Michael Muller
71f86c9970
Add VKey.restoreOfy() method for fixing ofy keys (#820)
Add a restoreOfy() instance method and a restoreOfyFrom() static method to
assist in restoring the objectify key for classes that have composite keys
that do not restore automatically.
2020-09-30 11:15:58 -04:00
Michael Muller
6f75dfd116
Create a flyway index file and verify correctness (#819)
* Create a flyway index file and verify correctness

Create an index file (flyway.txt) containing the names of all of the flyway
files and verify that it is ordered and in sync with the actual contents of
the flyway directory.  Also provide a target (generateFlywayIndex) to
automatically generate it.

The purpose of flyway.txt is to cause a merge conflict in the event that two
different developers add a flyway file with the same sequence number, an event
which has occurred multiple times.
2020-09-29 11:26:05 -04:00
Lai Jiang
ad5a74fee9
Revert "Request 101m CPU in sandbox proxy (#813)" (#818)
This reverts commit e30c0f9a11.

The proposed solution didn't work.
2020-09-25 11:55:46 -04:00
Lai Jiang
29b1ec4211
Add log4j-core as a runtime dependency (#817)
Without it we kept getting the following warning:

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
2020-09-24 19:59:39 -04:00
Weimin Yu
553d5717cb
Enhance the test for forbidden Schema changes (#815)
* Enhance the test for forbidden Schema changes

Current test is git-based. It is difficult to maintain and does not
catch out-of-order version numbers. It is also more aggressive than
necessary, failing on changes to submitted scripts that have not been
deployed yet.

The new test starts a database, deploys the current schema to it,
then deploys the set of Flyway scripts in this repository to the database.
2020-09-24 12:31:08 -04:00
Shicong Huang
1056fdbb64
Fix VKey reconstruction issue in BillingEvent (#805)
* Fix VKey reconstruction issue in BillingEvent

* Rebase on head
2020-09-23 19:04:58 -04:00