Commit graph

3557 commits

Author SHA1 Message Date
gbrodman
a07fbb27c5
Refactor PremiumList storage and retrieval for dual-database setup (#950)
* Refactor PremiumList storage and retrieval for dual-database setup

Previously, the storage and retrieval code was scattered across various
places haphazardly and there was no good way to set up dual database
access. This reorganizes the code so that retrieval is simpler and it
allows for dual-write and dual-read.

This includes the following changes:

- Move all static / object retrieval code out of PremiumList -- the
class should solely consist of its data and methods on its data and it
shouldn't have to worry about complicated caching or retrieval

- Split all PremiumList retrieval methods into PremiumListDatastoreDao
and PremiumListSqlDao that handle retrieval of the premium list entry
objects from the corresponding databases (since the way the actual data
itself is stored is not the same between the two

- Create a dual-DAO for PremiumList retrieval that branches between
SQL/Datastore depending on which is appropriate -- it will read from
and write to both but only log errors for the secondary DB

- Cache the mapping from name to premium list in the dual-DAO. This is a
common code path regardless of database so we can cache it at a high
level

- Cache the ways to go from premium list -> premium entries in the
Datastore and SQL DAOs. These caches are specific to the corresponding
DB and should thus be stored in the corresponding DAO.

- Moves the database-choosing code from the actions to the lower-level
dual-DAO. This is because we will often wish to access this premium list
data in flows and all accesses should use the proper DB-selecting code
2021-02-22 21:19:48 -05:00
Weimin Yu
ffe3124ee1
Properly set up JPA in BEAM workers (#965)
* Properly set up JPA in BEAM workers

Sets up a singleton JpaTransactionManger on each worker JVM for all
pipeline nodes to share.

Also added/updated relevant dependencies. The BEAM SDK version change
caused the InitSqlPipeline's graph to change.
2021-02-22 18:11:32 -05:00
Michael Muller
8f90b5746a
Fix ContactTransferData SQL loads (#967)
* Fix ContactTransferData SQL loads

ContactTransferData is currently loaded back from SQL as an unspecialized
TransferData object.  Replace it with the ContactTransferData object that we
use it to reconstitute.

It's likely that this could be done more straightforwardly with a schema
change.

* Changes requested in review
2021-02-22 13:42:20 -05:00
Ben McIlwain
a26905d39a
Note that immediate deletion is for exceptional circumstances only (#962)
* Note that immediate deletion is for exceptional circumstances only
2021-02-17 13:10:12 -05:00
Ben McIlwain
4ce790d29e
Fix obscure bug when checking restore prices of duplicate domain names (#968)
* Fix obscure bug when checking restore prices of duplicate domain names

There were instances of "java.lang.IllegalArgumentException: Multiple entries
with same key" in the logs, caused by attempting to construct an ImmutableMap
containing duplicate keys. It turns out this was happening in the domain check
flow when the following conditions were all simultaneously met:

1. The older v06 fee extension is used
2. The same domain name is being queried multiple times in a single check
   command (which is valid per the spec but doesn't actually make any sense)
3. Said domain exists
4. The cost of a restore (an uncommon operation) is being checked

When all of those conditions were met, an error was being thrown when the
dupe-containing list of domain names was used as the keys of a new Map. This
fixes that bug by calling .distinct() first.

Give enough registrars enough typewriters ...

BUG=179052195
2021-02-17 12:09:19 -05:00
sarahcaseybot
bcc1924b24
Refactor SignedMarkRevocationListDao for easy primary database cutover (#943)
* Refactor SignedMarkRevocationListDao for easy primary database cutover

* Fix javadoc comments

* Use PrimaryDatabase enum

* format fix

* fix up tests

* Fix punctuation

* Remove unnecessary else ifs

* Fix error messages

* spell out class name
2021-02-14 09:58:08 -05:00
Weimin Yu
f86936788e
Revert BEAM pipeline back to SQL credential file (#961)
* Revert BEAM pipeline back to SQL credential file

Stop using the SecretManager for SQL credential in BEAM for now. The
SecretManager cannot be injected into the code on pipeline workers
because RegistryEnvironment is not set.

See b/179839014 for details.
2021-02-11 14:06:13 -05:00
sarahcaseybot
13f61dd7b9
Add string constants for HTTP header names (#956)
* Add string constants for HTTP header names

* revert package-lock changes

* Clarify names

* add CONTENT_TYPE

* Fix formatting

* Move X-FORWARDED-FOR to ProxyHttpHeaders
2021-02-11 12:02:51 -05:00
Michael Muller
17cd9ba4f1
Add db-compare tests to three more flows (#963)
* Add db-compare tests to three more flows

Add database comparison to the replay tests for DomainDeleteFlowTest,
DomainRenewFlowTest and DomainUpdateFlowTest.
2021-02-11 11:35:13 -05:00
Weimin Yu
f983d564f8
Use Cloud SQL passwords from the Secret Manager (#959)
* Use Cloud SQL passwords from the Secret Manager

Continue fetching passwords from Keyring until this change is deployed.
2021-02-08 21:59:00 -05:00
sarahcaseybot
bdf9124e87
Add databaseTransitionSchedule entity and tool for updating (#926)
* Add databaseTransitionSchedule entitiy

* add UpdateDatabaseTransitionScheduleCommand

* small fixes

* change entity structure to no longer be singleton

* add get command

* fix getCache

* Change id to TransitionId enum

* more fixes

* Cleanup tests

* Add link to javadoc

* Add lastUpdateTime

* fix datatype of getCached
2021-02-08 17:22:00 -05:00
Ben McIlwain
d73e557acc
Rename some overlooked uses of "white-list" and "blacklist" (#960)
* Rename some overlooked uses of "white-list" and "blacklist"
2021-02-08 16:01:18 -05:00
Lai Jiang
0e74df82df
Update testcontainers to 1.51.1 (#958)
Per release note
(https://github.com/testcontainers/testcontainers-java/releases/tag/1.15.1),
this version is needed to support docker version 20.10 and above.
2021-02-08 14:31:17 -05:00
Lai Jiang
b59d2ae419
Upgrade to Gradle 6.8.2 (#957) 2021-02-08 09:35:43 -05:00
Weimin Yu
d7e65d95e6
Add a presubmit check to require use of templated SQL string literals (#954)
* Add a presubmit check to require use of templated SQL string literals

This PR proposes a coding style convention that helps prevent
SQL-injection attacks, and is easy to enforce in the presubmit check.

SQL-injections can be effectively prevented if all parameterized queries
are generated using the proper param-binding methods. In our project
which uses Hibernate exclusively, this can be achieved if we all follow
a simple convention: only use constant sql templates assigned to static
final String variables as the first parameter to creat(Native)Query
methods.

This PR adds a presubmit check to enforce the proposed rule, and
modified one class as a demo. If the team agrees with this proposal, we
will change all other use cases.
2021-02-06 19:28:38 -05:00
Michael Muller
1d96de98c9
Remove dual-write of registrar from tool commands (#952)
* Remove dual-write of registrar from tool commands

As discussed, we're keeping registrar in the "replicated" category.
2021-02-05 11:06:14 -05:00
Michael Muller
29bf0f3965
Make BiqueryPollJobAction endpoint internal only (#955)
* Make BiqueryPollJobAction endpoint internal only

This endpoint makes use of java object deserialization, which allows a
malicious actor to craft a request that can initiate overly broad actions on
the server.  Since this endpoint is not widely used for operational purposes,
limit its authorization to "internal only" so that no user agents (even with
admin privs) can access it.
2021-02-05 07:50:51 -05:00
sarahcaseybot
5100057dd5
Add cert enforcement in production start date (#953)
* Add start date for cert enforcement in production

* Add TODO to remove start date check after start date

* revert changes to package-lock.json

* Make start time a constant
2021-02-04 16:30:23 -05:00
Ben McIlwain
1e7260e022
Wire up DeleteExpiredDomainsAction so that it can actually be called (#951)
* Wire up DeleteExpiredDomainsAction so that it can actually be called

For now I'm just going to be calling it manually (and on sandbox for starters),
but in a few weeks, if all looks good, I'll add the cron job to regularly call
it in production, and this feature will thus be done.
2021-02-03 17:34:36 -05:00
Weimin Yu
73ba96a5d4
A better Datastore bulk delete tool (#947)
* A better Datastore bulk delete tool

Implements an alternative to GCP builtin tool with better performance
and usability.
2021-02-02 22:46:38 -05:00
Ben McIlwain
90db60643e
Add delete expired domains action (#836)
* Add delete expired domains action

This will be scheduled to run daily via App Engine cron in a subsequent PR.

* Add test
2021-02-02 18:46:15 -05:00
Ben McIlwain
98283a67ac
Use END_OF_TIME as sentinel value for domain's autorenewEndTime (#949)
* Use END_OF_TIME as sentinel value for domain's autorenewEndTime

Datastore inequality queries don't work correctly for null; null is treated as
the lowest value possible which is definitely the opposite of the intended
meaning here.

This includes an @OnLoad for backfilling purposes using the ResaveAll mapreduce.
2021-02-01 15:29:51 -05:00
Michael Muller
e70f14001c
Make cross database comparison recursive (#942)
* Make cross database comparison recursive

Cross-database comparison was previously just a shallow check: fields marked
with DoNotCompare on nested objects were still compared.  This causes problems
in some cases where there are nested immutable objects.

This change introduces recursive comparison.  It also provides a
hasCorrectHashCode() method that verifies that an object has not been mutated
since the hash code was calculated, which has been a problem in certain cases.

Finally, this also fixes the problem of objects that are mutated in multiple
transactions: we were previously comparing against the value in datastore, but
this doesn't work in these cases because the object in datastore may have
changed since the transaction that we are verifying.  Instead, check against
the value that we would have persisted in the original transaction.

* Changes requested in review

* Converted check method interfaces

Per review discussion, converted check method interface so that they
consistently return a ComparisonResult object which encapsulates a success
indicator and an optional error message.

* Another round of changes on ImmutableObjectSubject

* Final changes for review

Removed unnecessary null check, minor reformatting.

(this also removes an obsolete nullness assertion from an earlier commit that
should have been fixed in the rebase)

* Try removing that nullness check import again....
2021-01-29 18:57:20 -05:00
sarahcaseybot
22d3612be3
Convert Strings to X509 Certificates before validating (#948)
* Convert certificate strings to certificates

* Format fixes

* Revert "Format fixes"

This reverts commit 26f88bd313.

* Revert "Convert certificate strings to certificates"

This reverts commit 6d47ed2861.

* Convert strings to certs for validation

* Add clarification comments

* Add test to verify endoded cert from proxy

* Add some helper methods

* add tests for PEM with metadata

* small changes

* replace .com with .test
2021-01-29 16:59:57 -05:00
sarahcaseybot
ad8bc05877
Fix typo in header name in Client Certificate Provider (#946)
* Fix typo in header name

* fix test
2021-01-26 20:10:41 -05:00
Ben McIlwain
a3537447ef
Add clientCertificate to TlsCredentials.toString() (#945)
* Add clientCertificate to TlsCredentials.toString()

FlowRunner.run() logs these credentials to the GAE logs by implicitly using the
toString() method, so we need to add it if we want it to appear in the logs.
2021-01-26 17:20:21 -05:00
Ben McIlwain
4e66fed497
Use nullness parity helper (#944)
* Use nullness parity helper
2021-01-26 13:20:48 -05:00
gbrodman
886cdfa39b
Update NPM dependency based on Github security warning (#941) 2021-01-25 23:04:30 -05:00
sarahcaseybot
beefa9364b
Use CertificateChecker on login (#936)
* Use CertificateChecker on login

* Add actual enforcement of requirements in sandbox

* Add new Exceptions

* add validation command to RegistryToolComponent

* Fix error messages

* Add a test for production behavior

* check logs in test

* move loghandler
2021-01-22 16:32:15 -05:00
gbrodman
73210e4b09
Convert (most) HistoryEntry ofy calls to tm (#933)
* Convert (most) HistoryEntry ofy calls to tm

As part of this change, it was necessary to do changes in the JPATM that
are similar (but the opposite) of the changes that we did in
DatastoreTM with regards to converting HistoryEntries to and from the
*History classes.

We leave the ofy() calls in the MapReduce ResaveAllHistoryEntriesAction
for now; that can be converted during the Beam pipeline transition.

Some other tests required registrar-name fixes as well -- because
*History objects have a foreign key on the Registrar table, we have to
use a "real" registrar name in tests.

* Add simple HistoryEntryDaoTest
2021-01-22 14:43:34 -05:00
Ben McIlwain
08cec96a93
Correct containsMatch() -> contains() for non-regexes (#940)
* Correct containsMatch() -> contains() for non-regexes
2021-01-22 14:31:38 -05:00
Ben McIlwain
31ef402c50
Require an override flag to allow updating pending delete domains (#939)
* Require an override flag to allow updating pending delete domains

Needing to update pending delete domains is an uncommon situation, yet currently
we are allowing superusers to do so without any extra validation (which has led
to errors). This adds a new override flag to gate the update of pending delete
domains; without it, the update will fail.
2021-01-22 14:31:13 -05:00
Michael Muller
e89cc4406a
Fix another "extra parens" warning (#938)
* Fix another "extra parens" warning

Same place as the last one, but I missed it :-(
2021-01-22 13:39:30 -05:00
Shicong Huang
48de5d8375
Convert ofy() to tm() for all contact transfer flows (#937)
* Convert ofy() to tm() for all contact transfer flows

* Resolve comments
2021-01-22 09:38:51 -05:00
Ben McIlwain
59abc1d154
Put else if on same line to fix build style warning (#935)
* Put else if on same line to fix build style warning
2021-01-21 10:50:29 -05:00
Shicong Huang
6794c6fbd7
Resolve remaining TODO(shicong) (#932) 2021-01-20 19:27:48 -05:00
Ben McIlwain
0c384adc22
Change java.util.Optional.isEmpty() to !isPresent() (#934)
isEmpty() is not available in the version of Java GAE uses and is throwing
runtime errors (!!). I think these got into our codebases because people don't
have the language version set correctly in IntelliJ; they show as outright
errors for me (I'm on language level 8).
2021-01-20 09:38:52 -05:00
sarahcaseybot
3b679058b0
Validate Certificate on Login (#919)
* Check certificate matches saved one on login

* Add tests

* refactoring

* fix warning messages
2021-01-19 17:06:26 -05:00
Shicong Huang
9b5805f145
Resolve TODOs under Shicong's name (#930) 2021-01-15 14:22:31 -05:00
Michael Muller
9e6f99face
Add object comparison to replay tests (#925)
* Add object comparison to replay tests

Allow optional object comparison in the replay test extension and enable it
for the DomainCreateFlow test.

To faciliate this, add two new field annotations to ImmutableObject:
DoNotCompare, to be used for fields that are not relevant to either database,
and Insignificant, to be used for fields that are mutated after they have been
accessed and therefore violate immutability (there is currently only one of
these, however we might discover more in the course of adding more comparisons
to the replay test.

* Revert commented out premium price error log

* Added static create methods for ReplayExtension
2021-01-15 14:20:55 -05:00
Shicong Huang
554e675303
Add Sara and Legina to CONTRIBUTORS (#931)
* Add Sara and Legina to CONTRIBUTORS

* Add a new line at the end of the file
2021-01-14 20:16:55 -05:00
Michael Muller
3d33c81475
Remove some unnecessary parens (#929)
* Remove some unnecessary parens

Remove extra parens that cause a compile warning.

* Removed the correct paren this time.
2021-01-14 17:09:27 -05:00
Shicong Huang
56e384aa4f
Restore symmetric VKey referenced by BillingEvent.Cancellation (#928) 2021-01-14 13:16:24 -05:00
Shicong Huang
f669e3ca59
Convert remaining Contact flows to tm() (#924)
* Convert remaining Contact flows to tm()

* Add a test to verify street fileds get populated from XML
2021-01-13 13:50:23 -05:00
Shicong Huang
c45129f9ac
Convert some Host and Contact flows to tm() (#921)
* Convert HostDeleteFlow to tm()

* Add query for contact linked domain
2021-01-13 10:56:53 -05:00
Weimin Yu
84d2b82050
Update the Datastore to SQL migration pipeline (#927)
* Update the Datastore to SQL migration pipeline

The pipeline now includes all entity types to be migrated by it, and has
completed successfully using the Sandbox data set. The running time in Sandbox
is about 3 hours, extrapolating by entity count to a 12-hour run with
production data. However, actual running time is likely to be longer since
throughput is lower with domains, which accounts for a higher percentage
of the total in production. More optimization will be needed.

The migrated data has not been validated.
2021-01-12 18:05:46 -05:00
Shicong Huang
0109d5e473
Convert HostUpdateFlow to tm() (#923) 2021-01-08 10:28:22 -05:00
Ben McIlwain
9e03ae453c
Use better null-handling around registrar certificates (#922)
* Use better null-handling around registrar certificates

Now with Optional it's always very clear whether they do or do not have values.
isNullOrEmpty() shouldn't be necessary anymore (indeed it wasn't necessary prior
to this either, as the relevant setters in the Registrar builder already coerced
empty strings to null). And also the cert hash is a required HTTP header, so it
will error out in the Dagger component if null or empty long before getting to
any other code.

* Merge branch 'master' into optional-get-certs
2021-01-07 19:30:09 -05:00
Weimin Yu
7a62aa0602
Allow BEAM pipeline to choose JDBC isolation levels (#916)
* Allow BEAM pipeline to choose JDBC isolation levels

Some BEAM pipelines may only perform READ-ONLY (e.g., reporting) or
blind-write (datastore to sql data migration) operations, which do not
need the default TRANSACTION_SERIALIZABLE isolation level. In such
cases, a less strict level allows better performance.
2021-01-07 11:00:36 -05:00
Weimin Yu
6a1e86ff33
Add a TODO to a non-functioning JPA annotation (#917)
* Add a TODO to a non-functioning JPA annotation
2021-01-06 13:28:53 -05:00