Commit graph

3128 commits

Author SHA1 Message Date
Ben McIlwain
4cb88ab6e7
Convert RDE tests (and some test rules) from JUnit 4 to JUnit 5 (#677)
* Add JUnit Params and start using it

* Convert rest of RDE tests

* Don't check headers for generated tests

* Expand visibility to fix build breakage

* Bump JUnit versions to 5.6.2
2020-07-10 21:32:36 -04:00
gbrodman
987f390ff7
Run 'npm audit fix' to fix low-severity vulnerabilities in packages (#676) 2020-07-10 15:57:59 -04:00
Ben McIlwain
ca756e14e6
Migrate all model tests from JUnit 4 to JUnit 5 (#675)
* Make first handful of tests JUnit 5

* Migrate rest of model package to JUnit 5
2020-07-10 14:56:28 -04:00
Ben McIlwain
caa0cd9d61
Add a "coreDev" gradle target (#667)
* Add a "buildFmt" gradle target

This does the same thing as the automatic Java build target, except instead of
failing if the code formatting isn't correct, it just automatically reformats as
necessary and continues on.

* Remove unnecessary mustRunAfters

* Make it run tests too, and add :taskTree task

* Rename task to coreDev and remove run afters

* Add task tree dependency

* Actually that may not be necessary
2020-07-10 10:03:59 -04:00
Legina Chen
7806cc7edb
Add domainRepoId to Subdomain class (#674)
* Change Subdomain class to contain domainRepoId

* Remove jpaTm from Spec11PipelineTest and change clientId -> registrarId

* Remove 'client' from a comment

* Include changes to Spec11Pipeline

* add SafeBrowsingTransforms

* Run style
2020-07-09 16:26:35 -07:00
Lai Jiang
0964fdf1dc
Upgrade to Gradle 6.5.1 (#673) 2020-07-09 14:04:22 -04:00
gbrodman
d17ec1fcb1
Use an enum instead of boolean in EntityTestCase constructor (#669)
* Use an enum instead of boolean in EntityTestCase constructor

It's more clear to use an enum rather than just a simple boolean

* Add Javadoc and make the enum name more verbose
2020-07-09 12:54:32 -04:00
Ben McIlwain
fac5987c13
Double the # of pubapi instances to better handle traffic spikes (#671)
* Double the # of pubapi instances to better handle traffic spikes

We may also consider switching to an automatic scaling mode soon, on the hope
that it's working better than the last time we tried it (it would help to keep
resource costs down at least).
2020-07-09 11:52:15 -04:00
Ben McIlwain
a3319e0026
Upgrade flow test classes to JUnit 5 (#666)
Most of the diffs are visibility changes.

Also deletes ShardableTestCase, which was only necessary because of Blaze (and
possible Bazel) limitations.
2020-07-08 14:08:05 -04:00
Weimin Yu
5578464e06
Make sure uncommitted txn is rolled back (#665)
* Make sure uncommit txn is rolled back

The try block around commit that catches RuntimeException should also
catch Error, which is also unchecked.
2020-07-06 17:39:13 -04:00
gbrodman
c24a61f813
Refactor ContactResource into ContactBase and create ContactHistory (#634)
* Create ContactHistory class + table

This is similar to #587, but with contacts instead of hosts.

This also includes a couple cleanups for HostHistoryTest and RegistryLockDaoTest, just making code more proper (we shouldn't be referencing constant revision IDs when using a sequence that is used by multiple classes, and RLDT can extend EntityTest)

Note as well that we set ContactHistory to use the same revision ID sequence as HostHistory.

* Move ContactResource -> ContactBase

* Alter ContactBase and ContactResource
2020-07-06 12:52:16 -04:00
gbrodman
806f3b2456
Verify that the RegistryLock input has the correct registrar ID (#661)
* Verify that the RegistryLock input has the correct registrar ID

We already verify (correctly) that the user has access to the registrar
they specify, but nowhere did we verify that the registrar ID they used
is actually the current sponsor ID for the domain in question. This is
an oversight caused by the fact that our testing framework only uses
admin accounts, which by the nature of things have access to all
registrars and domains.

In addition, rename "clientId" to "registrarId" in the RLPA object

* Change the wording on the incorrect-registrar message
2020-07-05 22:31:14 -04:00
gbrodman
333170a724
Allow users the option of seeing their registry lock password (#663)
* Allow users the option of seeing their registry lock password

Only when entering it for the first time, of course.
2020-07-05 20:08:22 -04:00
Lai Jiang
47eeb8c4e4
Output PO number in detailed report (#659)
* Output PO number in detailed report

The PO number header was added during the beam migration but we forgot
to print the actual data in the corresponding column. This resulted in a
misalignment of columns in the detailed report.

This PR fixes it. Note that we cannot drop PO number from the header (as is not
useful in the detailed report) because the header represents all fields
that are to be parsed from the SQL query results, and PO number *is*
needed when generating the invoice itself. By dual-purposing the header
(both as the required fields in the parser and the first line in the
detailed report) we have to include the value of PO number in the
detailed report CSV as well.
2020-07-01 19:09:05 -04:00
Shicong Huang
391929b518
Expand AckPollMessagesCommand to ack PollMessage.Autorenew (#647)
* Expand AckPollMessagesCommand to ack PollMessage.Autorenew

* Rebase on master and address comment

* Resolve comments
2020-07-01 15:06:35 -04:00
gbrodman
7f62b7a89c
Include the registry lock email in the JS object as a sensitive field (#658)
* Include the registry lock email in the JS object as a sensitive field

* Change wording of exceptions to be more consistent
2020-07-01 13:05:21 -04:00
gbrodman
a1da32bfde
Disambiguate injected Cloud SQL parameter names (#657)
* Disambiguate injected Cloud SQL parameter names

This allows us to also inject the BeamJpaModule into RegistryTool, which
allows us to use the SocketJpaTransactionManager in Beam pipelines.

Some side effects of this include:
- duplication of KMS connections -- one standard, one Beam
- duplication of the creation of the partial Hibernate SQL configs
- removal of ambiguity between credentialFileName, credentialFilename,
and credentialFilePath -- we now use the latter.
- Performing the credential null check when instantiating the SQL
connection rather than when instantiating the module object. See the code
comments for more details on this.

I verified that this compiles and the tests run successfully when
injecting a @SocketFactoryJpaTm into a Beam pipeline.

* Remove two unnecessary config points and change the name of two params

* Use @Config instead of @Named and change the pool size

* Replace non-visible link with code
2020-07-01 11:55:21 -04:00
Weimin Yu
1961a5759d
Load Datastore snapshot from backup files (#660)
* Load Datastore snapshot from backup files

Defined a composite transform that loads from a Datastore export and
concurrent CommitLog files, identify entities that still exist at the
end of the time window, and resolve their latest states in the window.
2020-07-01 09:58:42 -04:00
Weimin Yu
d065ff63fc
Exclude Test/Monitoring Registrars from escrow (#655)
* Exclude Test/Monitoring Registrars from escrow

Registrars used for testing and monitoring should not be included
in Data escrow. They also lack the required ianaIdentifier property
and would fail ICANN data validation.

Note that since alpha and crash environments have bad data that
break the RDE process, we need to verify this change in Sandbox.
2020-06-26 19:11:22 -04:00
Michael Muller
07ff6279bb
Make EppResource.loadCached() use batched fetch (#652)
* Make EppResource.loadCached() use batched fetch

Use a batched fetch (ofy().load().keys(...)) from datastore in
EppResource.loadCached().

To support this, convert TransactionManager.load(Iterable<VKey>) to accept the
more flexible generic parameters and return a map.

* Simplify datastore key streaming

* Changes requested in review.
2020-06-26 13:50:02 -04:00
Legina Chen
5c5b6b20ce
Allow multiple threat types in the Spec11ThreatMatch table (#650)
* Update to generic Spec11ThreatMatch table

* Fix SQL syntax

* Make changes to the schema and add a test for null and empty threatTypes

* Fix a small typo

* Change the exception thrown with illegal arguments

Change the import for isNullOrEmpty

* Fix import for checkArgument

* Added a threat to test multiple threat types
2020-06-26 10:35:00 -07:00
Ben McIlwain
74b2de5c35
Make ImmutableMap Stream collect()ion nicer (#654)
This adds an entriesToImmutableMap() collector that can be used in place of
toImmutableMap(Map.Entry::getkey, Map.Entry::getValue()).

It also fixes up some existing calls that use toImmutableMap() when terser
alternatives exist.
2020-06-26 11:57:26 -04:00
Ben McIlwain
fba6804d3b
Properly handle restore fees on domain checks (#646)
* Properly handle restore command fees for domain checks

* Get tests working and handle fee classes better

* Remove unused ImmutableSet imports

* Fix code review concerns, mostly surrounding immutability

* Rename more targetIds and make them immutable too

* Merge remote-tracking branch 'upstream/master' into domain-check-restore-fees

* Fix Javadoc formatting
2020-06-26 10:59:46 -04:00
Lai Jiang
db5311075d
Patch terraform changes made internally (#651)
There were several LSC that made some formatting changes to our .tf
files. Export these changes externally for consistency.
2020-06-25 13:59:37 -04:00
Ben McIlwain
6e26dacdff
Make nomulus compatible with Java 11 (#649)
* Make nomulus compatible with Java 11

This fixes the double-spacing bug with logged EPP XML on App Engine that started
appearing after App Engine switching from using Java 8 to Java 11. Java 9 made
some changes to XML Transformer classes that needed a little bit of work to
accommodate.

This also fixes the unit tests that were failing in Java 11 (all of which were
related to said XML Transformer changes).

* Make code review changes
2020-06-25 13:17:22 -04:00
Weimin Yu
2e5466f32f
Fix flaky tests that depends on order of rules
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.

ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.

Added a mockito dependency, and upgraded mockito-core to 3.3.3.

Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
2020-06-24 23:14:47 -04:00
Lai Jiang
a65a3fd8b7
Add a property in Registry to disable invoicing (#644)
The added property will by default be null for existing entities and false for new entities. We could write a temporary MapReduce to set it to false for all existing ones, but it seems easier to just modify the query to account for the nullness of the column. The need to check for null won't exist once we migrate the Registry entity to Cloud SQL.

TESTED=deployed to alpha, set the property for tld zombo to true and triggered an export, verified that the column is null for all rows other than the row for zombo.
2020-06-24 16:21:38 -04:00
Weimin Yu
071800ae50
Delay SQL availability check during Tool start (#643)
* Delay SQL availability check during Tool start

During Cloud SQL migration, Cloud SQL password setup in Registry
server requires a few tool commands. Such commands must be able
to run before Cloud SQL is configured.

In this change, we deplay SQL credential loading until access is
needed.
2020-06-24 14:00:36 -04:00
Shicong Huang
75659671db
Fix a warning from Hibernate (#645)
Hibernate complained that ContactResource.getTransferData should not be
final:
WARN: HHH000305: Could not create proxy factory
for:google.registry.model.contact.ContactResource
org.hibernate.HibernateException: Getter methods of lazy classes cannot
be final: google.registry.model.contact.ContactResource#getTransferData
2020-06-24 10:02:42 -04:00
Michael Muller
c9d4ffb233
Convert all VKeys to symmetric VKeys (#641)
* Convert hosts to symmetric VKey

* Convert ContactResource to symmetric VKeys

* Convert BillingEvents to symmetric VKeys

* Converted PollMessage to symmetric VKeys

* Convert AllocationToken to symmetric VKeys

* Remove static methods, get everything working

* Changes requested in review.

* Removed newly introduced createOfy() calls
2020-06-24 08:02:11 -04:00
Weimin Yu
11ec4d64f2
Set up JpaTransactionManager in BEAM pipelines (#639)
* Set up JpaTransactionManager in BEAM pipelines

Added modules and utilities to create JpaTransactionManager in BEAM
pipelines.

Not wanting to set up AppEngine Remote API to access Keyring in the
Datastore, we instead use the credential files in GCS, which are
used by Spinnaker/Cloud Build and desktop access. Added utility
to download, decrypt, and parse the file. Also added/modified dagger
modules.
2020-06-23 11:04:52 -04:00
Ben McIlwain
e9ad1b6f72
Don't expect a renewal fee on restores when one isn't due (#637)
* Don't expect a renewal fee on restores when one isn't due

This is a fix on top of #632 so that domain restore commands don't require
acking an illusory renewal fee for 1 year when that isn't actually happening
(i.e. if the domain isn't yet past its original expiration).

Unfortunately, there's still a problem remaining wherein the restore fee on a
domain check will always include the additional year even if it's not
necessary. We don't have a good solution to that. Also note that in versions of
the fee extension more recent than 0.6, the fee extension cannot be passed on a
domain info command at all, so the domain check command is the only way you have
to determine what the restore fee should be. So we definitely do want to get
that right so that the apparent restore fee on a check is the same as the actual
restore fee when running the restore command. We're not quite there yet though
and it's hard to say how we will get there, since we don't load domains during a
domain check command for performance reasons yet we would need to do so in order
to know the expiration date and thus whether the additional year of renewal
should be charged.

A problem for a future PR.
2020-06-22 15:24:36 -04:00
Shicong Huang
e7db9b3c1a
Add a test to verify JPA supports polimorphic VKey (#642) 2020-06-22 15:13:38 -04:00
Legina Chen
ec09226baa
Create a Java entity to store ThreatMatch objects in SQL (#617)
* Squash everything together
Create SafeBrowsing_Threats table

Create LocalDateConverter and add indexes to SafeBrowsingThreats

Add indexes to SafeBrowsingThreats and make small style changes

Pass in DateTimeFormatter

Delete LocalDateConverterTest.java

Rebase

Make changes to ThreatType comments

Create LocalDateConverterTest

Add review changes

Add SafeBrowsingThreatTest

Rename repoId, refactor LocalDateConverterTest/SafeBrowsingThreatTest, add foreign keys

Change imports

Add foreign keys and rename version number

Add new generated db-schema file

Clean up null test cases

Add changes

Add foreign keys into SafeBrowsingThreatTeat and apply style checks

Add SafeBrowsingThreatTest into SqlIntegrationTestSuite and change golden file

Make small changes to SafeBrowsingThreatTest

Add tests for ForeignKeyViolations and remove setId in SafeBrowsingThreat

* Change V35 -> V36

* Add a foreign key test for a reference to Registrar

* Move some variables around
2020-06-22 12:07:59 -07:00
Weimin Yu
2c243a7d5f
Add testcontainers' Junit5 support dependency (#640)
* Add testcontainers' Junit5 support dependency

Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.

Migrated PersistenceModuleTest as an example.

Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
2020-06-22 13:12:58 -04:00
Ben McIlwain
df8ce38796
Rename whitelist -> allow list (#635)
* Rename whitelist -> allow list

* Merge branch 'master' into allowlist-denylist
2020-06-18 18:36:05 -04:00
Weimin Yu
382c8014de
Fix flakiness caused by ofy entity name conflict (#636)
* Fix flakiness caused by ofy entity name conflict

Overrode the 'kind' of two test entities to prevent name conflicts.
Tests are flaky because of this.

Added a check in AppEngineRuleTest for conflicting kinds.
2020-06-18 13:14:10 -04:00
Ben McIlwain
57113b4746
Show price of reserved domains when using matching allocation token (#632)
* Show price of reserved domains when using matching allocation token

When the registrar passes the fee extension, this shows the price of the domain
on a check command for reserved domains if the provided allocation token is a
match. Of course, the price is already always displayed on non-reserved names
(regardless of whether the specific provided token is a match or not).

This affects domain checks only; the price is already always displayed on domain
creates because you already by definition have access to register the domain in
question.
2020-06-18 11:57:22 -04:00
Weimin Yu
d43564172f
Refactor pipline for Datastore backup loading (#628)
* Refactor pipline for Datastore backup loading

Refactored pipeline transforms.

Added testing utilities that handles assertions better.

Investigated and documented challenges in serializing Ofy entities
without side effects.
2020-06-17 22:10:14 -04:00
gbrodman
69a1d04c18
Remove 'fullyQualified' from host and domain names (#631)
* Remove 'fullyQualified' from host and domain names

We don't actually enforce that these are properly fully-qualified
(there's no dot at the end) and we specifically use the term "label
name" when talking about labels.

Note: this doesn't convert FQDN -> DN (et al) in at least two types of
cases:
1. When the term is part of the XML schema
2. When the term is used by some external system, e.g. SafeBrowsing API

* Add TODO to rename fields
2020-06-17 16:19:26 -04:00
Michael Muller
19395def5b
Implement a persistable Transaction object (#614)
* Implement a persistable Transaction object

Implement Transaction, which encapsulates a sequence of datastore mutations
that can be serialized and written to the Cloud SQL Transaction table and
subsequently replayed to Datastore from a backend cron job.

* Changes requested in review

* Add a mujtation count to the persisted format
2020-06-17 14:16:48 -04:00
Shicong Huang
2f600e3e69
Implement remaining methods in JpaTransactionManager (#633) 2020-06-17 12:39:34 -04:00
Michael Muller
31841ccc55
Fix cookie processing for RDAP URL update (#630)
* Fix cookie processing for RDAP URL update

The existing code only does cookie processing on the _first_ Set-Cookie
header.  Therefore, if the "id" cookie used for authentication is defined in
anything other than the first Set-Cookie header (as it now is), we don't find
it.

Replace the cookie processing stanza with a line that processes all cookies in
all Set-Cookie headers.
2020-06-16 15:07:13 -04:00
gbrodman
4f37c65af5
Fix versioning semantic merge conflict (#629) 2020-06-16 12:51:28 -04:00
gbrodman
47178d4fb5
Add HostBase and HostHistory classes (#587)
* Add proof of concept for HostBase and HostHistory classes

* Use a PROPERTY accessor for @Ids

* Add an unused setter method for Hibernate's sake

* Refactor HostHistory

* Some responses to CR

* Fix relationship and test

* Manually manage the foreign keys for HostHistory

* Protect HostBase's builder and use text for the enum type

* Add responses to CR

- Add javadocs
- Create an ID sequence for host history objects

* Don't try to set the ID

* Use a Long and remove the setter

* Add some comments and rename a couple fields

* Don't change Datastore schema

* Use Long in the Datastore schema

* Add new createVKey method

* Add comments and rename fields

* Rename v27->v31 and regenerate the golden

* Fix superordinateDomain and inetAddresses in HostHistory

* V31 -> V32

* Fix SQL files that got messed up in the merge

* Configure and use a manually-created history ID sequence

* Add three more indices to HostHistory
2020-06-16 11:47:17 -04:00
Shicong Huang
26e2a51180
Refactor TransferData to remove unused fields in Contact table (#623)
* Add DomainTransferData and ContactTransferData

* Refactor TransferData to remove unused fields in Contact table

* Add scope for TransferData's type parameter
2020-06-16 10:42:57 -04:00
Michael Muller
21f2f38ad1
Allow class-specific creation of symmetric VKeys (#625)
* Allow class-specific creation of symmetrict VKeys

When translating from a datastore Key to a VKey, see if the "kind" class
contains a createVKey(com.googlecode.objectify.Key) static method and if it
does, use it to construct a symmetric VKey instead of simply creating an
objectify-sided asymmetric VKey.

As a test case for this, implement the createVKey() static function for
DomainBase.  Also, create unit tests for VKeyTranslatorFactory, which
continues to house the functionality.
2020-06-15 11:35:03 -04:00
Lai Jiang
0e04f6ca5b
Upgrade to Gradle 6.5 (#627) 2020-06-15 10:57:29 -04:00
Ben McIlwain
4be70c8509
Refactor Fee handling so that each fee knows if it's premium (#626)
* Refactor Fee handling so that each fee knows if it's premium

This is a noop for now, as the new isPremium boolean isn't yet used by anything,
but it will be used in follow-up PRs to add additional fee information using the
fee extension (see: b/157621273).  Specifically what we're trying to do here is
return <fee:command name="create" standard="1"> (using the finalized version of
the fee extension) when an entire command has no premium fee associated with
it. And in the current earlier versions of the fee extension that we support,
we'll want to display the correct fee amount and class for creates/checks on
reserved domains when a valid allocation token is passed. This also needs the
isPremium information.

There are no testing implications yet because isPremium isn't exposed anywhere,
but there will definitely be lots of test changes once it's feeding into EPP-
visible changes.

* Rename things, add method Javadoc

* Apply formatting
2020-06-12 16:43:02 -04:00
Ben McIlwain
cf1448bca8
Restore the original expiration time on domain restore (#601)
* Restore the original expiration time on domain restore

Except if that time is now in the past, then add a year to it.

* Apply auto-formatter changes to fix my local build

* Merge branch 'master' into restore-expiry-date

* Fix reversed comments
2020-06-12 14:33:49 -04:00