Commit graph

3278 commits

Author SHA1 Message Date
Shicong Huang
7a68b1b6f0
Revert package-lock.json to version from #676 (#834)
Co-authored-by: gbrodman <gbrodman@google.com>
2020-10-09 15:57:23 -04:00
Shicong Huang
14e593d9e1
Add SchemaCrawler as dependency (#833) 2020-10-09 15:02:11 -04:00
Weimin Yu
2d5de96fbd
Minor python changes (#832)
* Minor python changes

Use dataclasses instead of attrs. The former is part of the standard lib
while the latter may need to be installed separately.

Also added python3 to the list of prerequisites.
2020-10-09 14:50:21 -04:00
Weimin Yu
13d30b0bfb
Maintain a release-to-Version map in deployment (#831)
* Maintain a release-to-Version map in deployment

Keep track of the mapping between Nomulus release tags and AppEngine
version ids with a mapping file. This is necessary because AppEngine
does not support custom versioning. With this mapping, rollbacks could
be automated. Automation of rollbacks is important since there are
test-supporting metadata to be updated, but are easily forgotten.

During the last stage of deployment, current per-service version ids
are fetched using gcloud and are appended to a file on GCS. Each line
is of the format "{RELEASE_TAG},{APPENGINE_SERVICE},{APPENGINE_VERSION}.

This change has been tested in crash. The rollback script is still a
work in progress.
2020-10-09 13:32:52 -04:00
Shicong Huang
b05f6b4ba3
Add SQL schema for DelegationSignerData (#713)
* Add SQL schema for DelegationSignerData

* Remove join table

* Rebased on HEAD

* Rebase on head
2020-10-09 10:22:31 -04:00
Shicong Huang
17a1387184
Disable auto-generation on id for HostHistory and ContactHistory (#827) 2020-10-08 12:30:54 -04:00
Ben McIlwain
2e230664fd
Convert CertificateViolation into an enum (#829)
* Convert CertificateViolation into an enum

This ends up being nicer to deal with from callsites than class instances, while
still permitting full configurability of all parameters. There are various other
changes/fixes as well.
2020-10-07 22:19:36 -04:00
Michael Muller
299b093f78
Correctly restore composite VKeys in DomainContent (#825)
* Restore composite vkeys in DomainContent

PollMessage/BillingEvent vkeys in DomainContent must have their ofy keys
restored from other fields in DomainContent (namely the repo id and their
specific history event ids).

Add PostLoad methods to DomainContent and DomainHistory to do the restoration.

* Fixes for review.

* Deal with foreign-key cycles
2020-10-07 12:42:01 -04:00
Ben McIlwain
61e7fa89f7
Fix incorrect repackaged App Engine import (#828)
* Fix incorrect repackaged App Engine import
2020-10-07 11:52:17 -04:00
sarahcaseybot
6ab69d4226
Add a CertificateChecker class (#793)
* CertificateChecker with checks for expiration and key length

* Add validity length check

* Get rid of hard-coded constants and DSA checks

* add files that for some reason weren't included in last commit

* Rename violations and other fixes

* Add displayMessage to CertificateViolation enum

* Switch violations from an enum to a class

* small changes

* Get rid of ECDSA checks

* add checks for old validity length

* Change error message for validity length
2020-10-06 15:47:42 -04:00
gbrodman
0f09a4a0ab
Add more Datastore/Sql Entity annotations (#826)
* Add more Datastore/Sql Entity annotations

* Move comments up a line
2020-10-05 13:07:53 -04:00
Shicong Huang
95f6ccc657
Fix vkey reconstruction for PollMessage (#823)
* Fix vkey reconstruction for PollMessage

* Add foreign key

* Rebase on HEAD
2020-10-05 10:35:40 -04:00
Michael Muller
77fabe4dc4
Move "WithLongVKey" to BillingEvent subclasses (#821)
When loading the VKeys for the BillingEvents hierarchy, it is necessary to
restore the original concrete class for the type, otherwise we end up with a
different (and incompatible) VKey.

As part of this, convert the cancellation matching billing event to
VKey<Recurring>, which seems like the only thing it actually can be.
2020-10-02 15:20:23 -04:00
Lai Jiang
71fa12f773
Fix invoicing SQL (#824) 2020-10-01 14:29:49 -04:00
Shicong Huang
fd40a6a2b9
Use composite primary key for HostHistory and ContactHistory (#809)
* Use composite primary key for HostHistory and ContactHistory

* Update flyway file version

* Make getters private

* Add javadoc

* Rebase on HEAD
2020-10-01 11:01:57 -04:00
Michael Muller
71f86c9970
Add VKey.restoreOfy() method for fixing ofy keys (#820)
Add a restoreOfy() instance method and a restoreOfyFrom() static method to
assist in restoring the objectify key for classes that have composite keys
that do not restore automatically.
2020-09-30 11:15:58 -04:00
Michael Muller
6f75dfd116
Create a flyway index file and verify correctness (#819)
* Create a flyway index file and verify correctness

Create an index file (flyway.txt) containing the names of all of the flyway
files and verify that it is ordered and in sync with the actual contents of
the flyway directory.  Also provide a target (generateFlywayIndex) to
automatically generate it.

The purpose of flyway.txt is to cause a merge conflict in the event that two
different developers add a flyway file with the same sequence number, an event
which has occurred multiple times.
2020-09-29 11:26:05 -04:00
Lai Jiang
ad5a74fee9
Revert "Request 101m CPU in sandbox proxy (#813)" (#818)
This reverts commit e30c0f9a11.

The proposed solution didn't work.
2020-09-25 11:55:46 -04:00
Lai Jiang
29b1ec4211
Add log4j-core as a runtime dependency (#817)
Without it we kept getting the following warning:

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
2020-09-24 19:59:39 -04:00
Weimin Yu
553d5717cb
Enhance the test for forbidden Schema changes (#815)
* Enhance the test for forbidden Schema changes

Current test is git-based. It is difficult to maintain and does not
catch out-of-order version numbers. It is also more aggressive than
necessary, failing on changes to submitted scripts that have not been
deployed yet.

The new test starts a database, deploys the current schema to it,
then deploys the set of Flyway scripts in this repository to the database.
2020-09-24 12:31:08 -04:00
Shicong Huang
1056fdbb64
Fix VKey reconstruction issue in BillingEvent (#805)
* Fix VKey reconstruction issue in BillingEvent

* Rebase on head
2020-09-23 19:04:58 -04:00
Lai Jiang
4aaf31be9f
Update IDN tables per ICANN's request (#812)
See b/168508962 for the request.
2020-09-21 23:08:10 -04:00
Lai Jiang
e30c0f9a11
Request 101m CPU in sandbox proxy (#813)
This is suggested as a mitigation to allow us to deploy to sandbox. The default
value is 100m.

See: https://b.corp.google.com/issues/167295064#comment36.
2020-09-21 21:22:41 -04:00
gbrodman
2a5d9c8ef5
Allow explicitly for null EPP resources in History objects (#790)
* Allow explicitly for null EPP resources in History objects

* Repo IDs should always be nonnull

* Add a test to verify loading / comparison of legacy HistoryEntry objects

* Format javadoc + annotations

* More javadoc changes

* V52 -> V56

* V56 -> V57

* saveNew -> insert in new tests
2020-09-21 15:50:15 -04:00
gbrodman
597f5746a4
Rename V54 -> V56 on host table rename (#811) 2020-09-21 13:34:37 -04:00
Michael Muller
5bff53a711
Rename HostResource table to Host (#804)
* Rename HostResource table to Host

* Convert drop/creates to renames
2020-09-21 11:43:25 -04:00
Michael Muller
933394e8c3
Improve naming of TransactionManager methods (#802)
* Improve naming of TransactionManager methods

Per internal discussion, convert names of methods as follows:

    saveNew -> insert
    saveNewOrUpdate -> put
    checkExists -> exists

Likewise, convert derived names to their corresponding forms, e.g.
saveNewOrUpdateAll -> putAll.
2020-09-21 09:10:01 -04:00
gbrodman
3b841bbb5b
Add domain-specific history fields to DomainHistory objects (#794)
* Add domain-specific history fields to DomainHistory objects

* Add javadoc for Hibernate-only methods

* V52 -> V54

* Use only a single DomainTransactionRecord table

* Add nullables and fix up a comment

* V54 -> V55

* Regenerate db schema

* Regen SQL file
2020-09-18 15:55:17 -04:00
gbrodman
798879d031
Fix semantic merge conflict in Registry (#810)
* Fix semantic merge conflict in Registry
2020-09-18 14:40:11 -04:00
gbrodman
1a63d50b82
Create a separate per-tld registry lock/unlock cost (#800)
* Create a separate per-tld registry lock/unlock cost

Currently we use the standard server status change cost for this, but
this might not be ideal at some point in the future if we wish to allow
manual forced updates outside of the standard registry lock system (we
would charge for these manual forced updates, even if we don't charge
for registry locks).

* Remove period
2020-09-18 13:22:29 -04:00
Lai Jiang
054571a625
Update ICANN activity reporting logging (#808)
1. It appears that when we have a 200 response, the response content is
   garbled, but we don't care since we know the request is successful.
   When we have a 400 response, the response is indeed UTF-8 encoded.
   Print the stack trace of the HTTP exception doesn't help anymore.

2. The result code is a complex type which includes the xml element with
   all its attributes, which don't care. We only want to print the
   value.
2020-09-18 11:37:23 -04:00
sarahcaseybot
7468a9915b
Migrate Registry objects to a TLD table in Cloud SQL (#803)
* Add TLD table

* Change reservedLists to array

* Change ReservedLists back to a set

* Rename reservedListKeyConverter to ReservedListKeySetConverter

* Add a postload method
2020-09-17 12:47:50 -04:00
Lai Jiang
157d9f75c1
Use the save API version for the HPA controller and the deployment (#807)
This supposedly can fix our deployment problem. Tested on alpha.

Also updated the deployment script to replace the service object as
well.
2020-09-16 09:15:35 -04:00
Lai Jiang
b2e4f07bb9
Update IDN tables (#806)
See b/168508962 for context.

1. Changed the contact to iana-contact@google.com
2. Changed the header from "Script" to "Language" for zh-Hans and
   zh-Hant.
3. Commented out the references in zh-Hans and Zh-Hant
2020-09-16 09:15:27 -04:00
Michael Muller
5488e1b323
Fix accessing superclass fields in checkExists() (#799)
* Fix accessing superclass fields in checkExists()

JpaTransactionManagerImpl doesn't respect @Id fields in mapped superclasses.
Replace calls to getDeclaredId() and getDeclaredField() with superclass
friendly counterparts.
2020-09-11 13:45:51 -04:00
Shicong Huang
5ab0f97351
Add and use temp_history_id_sequence to avoid release error (#795) 2020-09-11 12:25:08 -04:00
sarahcaseybot
f7b65327da
Add type converter for Key<ReservedList> and Key<PremiumList> (#796)
* Add converter for reservedlist and premiumlist keys

* Remove public modifier from test classes
2020-09-10 17:36:22 -04:00
Michael Muller
36482ce94f
Fix the billing occurrence foreign key (#797)
* Fix the billing occurrence foreign key

Fix the Domain.billing_occurrence_id foreign key constraint to reference the
correct table (BillingRecurrence, not BillingEvent).
2020-09-10 12:02:24 -04:00
Lai Jiang
125f509b46
Change disable invoicing flag to enable invoicing flag (#783)
* Change disable invoicing flag to enable invoicing flag

This flag will be the sole determinor on if invoicing is enabled,
regardless of TLD types.

Once this PR is deployed we will need to run the nomulus command to
update this flag on all launched open TLDs.

For context on why this change is made, see b/159626744.

* Rename enableInvoicing to InvoicingEnabled
2020-09-09 19:37:41 -04:00
gbrodman
fb7ba80b86
Implement DatastoreEntity/SqlEntity for many more classes (#788)
* Implement DatastoreEntity/SqlEntity for many more classes

We still have many more classes to go, but this gets us closer to
guaranteeing that we can convert from Datastore to SQL objects and back
again.

* Shift SqlEntity impl to HistoryEntry
2020-09-09 13:56:59 -04:00
Lai Jiang
a86fcf79f7
Make ICANN reporting not fail on success upload (#791)
* Make ICANN reporting not fail on success upload

According to the spec
(https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-13#page-16),
when an upload succeeds (HTTP response code 200), the result code
contained in the response message is always 1000 (success). So there is
no need to parse the response content and check the result code. Given
that we are having a problem parsing the response content due to encoding,
it is best that we don't check it so as to not get false negative
alerts when the upload is successful.

The current logic also has a bug: HttpRequest.execute() will by default
throw when the response code is non-20X. Therefore for a 400 response,
our parsing logic never runs on it. Coincidentally, this month when we
uploaded the July activity report (due to stale cursors), we get 400
responses (due to existing reports on the ICANN servers). The stack
trace printed for the thrown exceptions from the 400 responses contained
correctly parsed response contents. This lead us to believe that the issue with
encoding was transient last month. However when we tried again to upload this
month's report, our parser failed again (because the response code was 200 this
time, and our parser actually ran on the response contents).

This seems to suggest that ICANN is sending back readable response
contents, but our parser somehow failed to understand it, assuming that
ICANN is using the same encoding for 200 (which we tried and failed to
parse) and 400 response contents (which caused an exception and was printed
corrected in the stack trace).

This PR changed the transport behavior so that it doesn't throw
automatically for non-20X responses. We will print the content for both
200 and 400 responses, but only try to parse 400 response content. We
put the 400 response in an HttpResponseException and print stack trace
from it, which should display the content correctly so that we can
compare it with the result of our own parsing.

* Add tests
2020-09-03 15:57:30 -04:00
Lai Jiang
dc8e095e55
Upgrade to Gradle 6.6.1 (#792) 2020-09-03 15:56:52 -04:00
Shicong Huang
cdf2c7f7cb
Merge ClaimsList into ClaimsListShard (#694)
* Merge ClaimsList into ClaimsListShard

* Add a TODO to rename the class

* Rebase on HEAD

* Improve javadoc
2020-09-03 11:18:40 -04:00
Shicong Huang
ecafebdc3d
Use composite primary key for DomainHistory (#767)
* Use composite primary key for DomainHistory

* Move History table's SequenceGenerator to orm.xml

* Rebase on HEAD and remove default value for key in History tables

* Use primitive type for id.

* Revert the cache change
2020-09-03 10:21:23 -04:00
Lai Jiang
c6c8d21281
Update jackson-core to the latest version (#789)
Vomit identified a vulnerability in the current version.
2020-09-03 09:11:12 -04:00
Shicong Huang
5f6ea2cbf2
Fix cascade issue for GracePeriod (#775)
* Fix cascade issue for GracePeriod

* Rebase on HEAD

* Make GracePeriod immutable

* Add javadoc and use nullToEmptyImmutableCopy
2020-09-02 20:05:53 -04:00
Shicong Huang
393c388e0d
Consolidate conversion from Duration to Period in DurationConverter (#786)
* Consolidate conversion from Duration to Period in DurationConverter

* Resolve comment
2020-09-01 11:29:28 -04:00
gbrodman
5a08ce498e
Revert "Change the wording on the lock-not-enabled page (#504)" (#787)
This reverts commit 28d3af0ee9.

We are now ready to accept new Registry Lock requests so we can have the
originally-designed wording back in place
2020-08-31 15:19:42 -04:00
Weimin Yu
5db8cbc994
Fix flaky web driver tests (#784)
* Fix flaky web driver tests

Identified two flaky tests in RegistrarConsoleScreenshotTest through
local testing and fixed them by waiting for specific web elements instead
of using fixed delays.

Refactored the wait methods to support different test scenarios,
and removed unnecessary delays.

Extensively tested locally. Also ran multiple presubmits on Kokoro.
2020-08-31 15:09:54 -04:00
Weimin Yu
bbcafea98e
Cover more base in forbidden SQL change check (#785)
* Cover more base in forbidden SQL change check

Update the forbidden SQL change detection script to include file deletion and
renaming as well as edits.
2020-08-31 15:08:37 -04:00