Commit graph

3629 commits

Author SHA1 Message Date
Lai Jiang
4246e7e4e0
Add indexes on contacts in the Domain table (#1145)
These indexes are used to find if a contact is linked to a domain in
during a contact delete.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1145)
<!-- Reviewable:end -->
2021-05-13 10:47:35 -04:00
Lai Jiang
9f21989f13
Remove the logic to add full certificate in the headers (#1143)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1143)
<!-- Reviewable:end -->
2021-05-12 20:52:16 -04:00
gbrodman
2073f5b59f
Populate the host in HostHistory objects in Host flows (#1129)
* Populate the host in HostHistory objects in Host flows
2021-05-12 19:11:30 -04:00
Weimin Yu
66ac000ef4
Fix the JPA Read connector for large data (#1155)
* Fix the JPA Read connector for large data

Allow result set streaming by setting the fetchSize on JDBC statements.
Many JDBC drivers by default buffers the entire result set, causing
delays in first result and/or out of memory errors.

Also fixed a entity instantiation problem exposed in production runs.

Lastly, removed incorrect comments.
2021-05-12 19:07:38 -04:00
Rachel Guan
85bac9834f
Add stageEntityChange() method to display difference when creating a reserved list (#1149)
* Add stageEntityChange() method to display difference before execution when creating a reserved list
2021-05-12 17:32:57 -04:00
Weimin Yu
484e30cd80
Restore a fix for flaky test (#1154)
* Restore a fix for flaky test

Restore a speculative fix for the flakiness in
DeleteExpiredDomainsActionTest. Although we identified a bug and fixed
it in a previous commit, it may not be the only bug. The removed fix may
still be necessary.
2021-05-12 16:03:42 -04:00
gbrodman
af67356aa0
Convert more ofy() to auditedOfy() calls (#1152)
A couple of these use the QueryComposer interface to avoid branching.

In addition, we enforce the Datastore restriction that there can be at
most 1 field with an inequality query, see https://cloud.google.com/appengine/docs/standard/go111/datastore/query-restrictions#inequality_filters_are_limited_to_at_most_one_property
2021-05-12 15:06:19 -04:00
Rachel Guan
8c9a2b5f4a
Fix typo in comment of premium list example file (#1148)
* Fix typo in comment of premium list example file
2021-05-11 18:25:05 -04:00
gbrodman
0d67ea3a6e
Combine the two Lock classes into one class (#1126)
* Combine the two Lock classes into one class

This allows us to remove the DAO and to just treat locks the same as we
would treat any other object -- generically grabbing them from the
transaction manager.

We do not need to be concerned about the changeover between Datastore
and SQL because we assume that any such changeover will require
sufficient downtime that any currently-valid acquired locks will expire
during the downtime. Otherwise, we could get into a situation where an
action has acquired a particular lock in Datastore but not SQL.
2021-05-11 16:37:40 -04:00
Rachel Guan
5b56e8b71b
Create key based on the change type (#1147)
* Create key based on the change type
2021-05-11 15:24:35 -04:00
Weimin Yu
6eba8aa1c4
Fix timestamp inversion bug (#1144)
* Fix timestamp inversion bug

Set the number of commitLog buckets to 1 in CommitLog replay tests to
expose all timestamp inversion problems due to replay. Fixed
PollAckFlowTest which is related to this problem.

Also fixed a few tests that failed to advance the fake clock when they
should, using the following approaches:

- If DatabaseHelper used but clock is not injected, do it. This
  allows us to remove some unnecessary manual clock advances.
- Manually advance the clock where convenient.
- Enable clock autoIncrement mode when calling production classes that
  performs multiple transactions.

We should consider making 1-bucket the default setting for tests. This
is left to another PR.
2021-05-11 14:51:10 -04:00
Lai Jiang
8d18450e56
Update README.md (#1146)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1146)
<!-- Reviewable:end -->
2021-05-11 13:40:07 -04:00
sarahcaseybot
65be65fb24
Always use Cloud SQL as primary for ClaimsList (#1127)
* Always use Cloud SQL as primary for ClaimsList

* Add a test back
2021-05-10 16:47:34 -04:00
Weimin Yu
984f1118e3
Make secretmanager primary storage for keyring (#1124)
* Make secretmanager primary storage for keyring

Also removed the migrate_kms_keyring command.
2021-05-10 11:11:26 -04:00
gbrodman
0bcb142bc9
Add an auditedOfy marker method for allow-listed ofy() calls (#1138)
* Add an auditedOfy marker method for allow-listed ofy() calls

This will allow us to make sure that every usage of ofy() has been
hand-examined and specifically allowed.
2021-05-10 10:55:28 -04:00
Lai Jiang
d93a4e562a
Delete hosts synchronously when using SQL (#1141)
Also put some common logic in helper funcions in ContactDeleteFlowTest
to reduce clutter.
2021-05-10 10:22:01 -04:00
Lai Jiang
420a579e01
Fix flaky Spec11PipelineTest (#1133) 2021-05-07 15:01:11 -04:00
Lai Jiang
1ec96b66e2
Perform synchronous contact delete in SQL (#1137)
In SQL the contact of a domain is an indexed field and therefore we can
find linked domains synchronously, without the need for MapReduce.

The delete logic is mostly lifted from DeleteContactsAndHostsAction, but
because everything happens in a transaction we do not need to recheck a
lot of the preconditions that were necessary to ensure that the async
delete request still meets the conditions that when the request was
enqueued.
2021-05-07 10:48:51 -04:00
gbrodman
51a7ba249e
Populate the contact in ContactHistory objects created in Contact flows (#1111)
* Populate the contact in ContactHistory objects created in Contact flows

Minimal interesting changes here
- a bit of reconstruction in ContactHistory to get the repo ID from the
key
- making the History revision ID Long instead of long so that it can be
null in non-built intermediate entities
- adding a copyFrom(HistoryEntry.Builder) method in HistoryEntry.Builder
so that we don't need to allocate quite as many unnecessary IDs, i.e.
removing the .build() lines in provideContactHistory and
provideDomainHistory
2021-05-06 14:38:55 -04:00
Lai Jiang
5120397607
Upload the GCB delete job yaml file to GCS (#1135)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1135)
<!-- Reviewable:end -->
2021-05-05 21:43:51 -04:00
sarahcaseybot
038825f254
Always use Cloud SQL as primary for Reserved and Premium Lists (#1113)
* Always use Cloud SQL as primary for Reserved and Premium Lists

* small typos

* Add a state check

* Add test for bloom filter

* fix import
2021-05-05 17:24:06 -04:00
Weimin Yu
b38574a9fc
Add a BEAM read connector for JPA entities (#1132)
* Add a BEAM read connector for JPA entities

Added a Read connector to load JPA entities from Cloud SQL.

Also attempted a fix to the null threadfactory problem.
2021-05-05 15:45:03 -04:00
Lai Jiang
3f6ec8f1b0
Re-enable tests in RC build (#1130)
There has been a case where the CI was broken on Friday and no one
noticied or fixed it and a RC build was built with broken tests.
The tests were disabled due to unknown test failures that have since
been fixed.

Also update the machine type used by GCB to be more powerful. This is
necessary for the tests to past because N1_HIGHCPU_8 is RAM constraint
and the tests crashes. I updated all jobs to use the new type which
hopefully will make the build faster as well.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1130)
<!-- Reviewable:end -->
2021-05-05 13:53:21 -04:00
gbrodman
65fb0c6cff
Update Karma version to avoid security hole in dependency (#1134)
This also forces the karma test to use the Gradle-installed version of
node instead of the global version. The global version installed on the
Kokoro machines is too old to function with some of the newer libraries.
2021-05-05 13:50:45 -04:00
Lai Jiang
e63085fb6a
Add a GCB job to delete stopped GAE versions (#1128) 2021-05-05 11:27:46 -04:00
gbrodman
b5363e9457
Populate the domain in DomainHistory objects created in Domain flows (#1106)
Unfortunately, much of the time there's a bit of a circular dependency
in the object creation, e.g. the Domain object stores references to the
billing events which store references to the history object which
contains the Domain object. As a result, we allocate the history
object's ID before creating it, so that it can be referenced in the
other objects that store that reference, e.g. billing events.

In addition, we add a utility copyFrom method in HistoryEntry.Builder to
avoid unnecessary ID allocations.
2021-05-04 19:09:27 -04:00
Ben McIlwain
cb16df235a
Remove unnecessary MockitoExtension from Spec11PipelineTest (#1115)
* Remove unnecessary MockitoExtension from Spec11PipelineTest

This is kind of a shot in the dark here, but this is one of the obvious
differences between this test class (which frequently experiences flakes) and
the other pipeline test classes which do not.

It's also possible we were getting the wrong runner if the test framework was
incorrectly detecting an App Engine runtime environment, so I added an assert
that will make it very clear if this is the cause of any failures.
2021-05-04 18:38:24 -04:00
Lai Jiang
d285edef3d
Fix a few linter warnings (#1122) 2021-05-04 13:35:31 -04:00
Weimin Yu
509c0dcd17
Handle bad production data when migrating to SQL (#1120)
* Handle bad production data when migrating to SQL

Ignore or fix bad entites when populating SQL with production data in
Datastore. These are mostly inconsistent foreign keys.

See b/185954992 for details.
2021-05-03 16:09:43 -04:00
sarahcaseybot
ce18bf0690
Use FakeClock to prevent Expired Certificate Violations (#1121)
* Use FakeClock to prevent Expired Certificate Violations

* Format fixes

* Make CertificateChecker static
2021-05-03 15:10:26 -04:00
Lai Jiang
8d63cbfca0
Remove enforcement date from the SslServerInitializer (#1117)
The enforcement data has passed and ICANN has confirmed that their web
WHOIS prober conforms to our requirements.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1117)
<!-- Reviewable:end -->
2021-04-30 15:44:03 -04:00
Lai Jiang
eb6a1fe1ed
Remove Pipeline as a field in pipeline classes (#1119)
In tests we use a TestPipelineExtension which does some static
initialization that should not be repeated the same JVM. In our
XXXPipeline classes we save the pipeline as a field and usually write lambdas
that are pass to the pipeline. Because lambdas are effectively anonymous inner
classes they are bound to their enclosing instances. When they get serialized
during pipeline execution, their enclosing classes also do. This might result
in undefined behavior when multiple lambdas in the same XXXPipeline are used
on the same JVM (such as in tests) where the static initialization may be done
multiple times if different class loaders are used. This is very
unlikely to happen but as a best practice we still remove them as
fields.
2021-04-30 14:32:33 -04:00
Weimin Yu
431710c95b
Improve usability of WipeOutCloudSqlAction (#1118)
* Improve usability of WipeOutCloudSqlAction

Replace the "drop owned" statement with ones that drops only tables and
sequences. The former statement also drops default grants for the
nomulus user, which must be restored before the database can be used by
the nomulus server and tools.
2021-04-29 23:09:20 -04:00
Michael Muller
1fdf9cb979
Convert GenerateLordnCommand to tm (#1091)
* Convert GenerateLordnCommand to tm

This makes use of QueryComposer and adds a `list()` method to it.

Since there was no test for GenerateLordnCommand, this also implements one.

* Changes requested in review

* Add test for list queries

* Stream domains instead of listing them

* Reformatted
2021-04-29 13:14:56 -04:00
Michael Muller
95fdd36c77
Make nom_build not check for ".git" directory (#1110)
* Make nom_build not check for ".git" directory

nom_build tries to verify that it is in the root of the tree prior to doing
anything, however checking for a .git directory doesn't work in a merged
directory.

* Minor formatting fix to attempt to force rebuild
2021-04-28 11:23:39 -04:00
Ben McIlwain
d239a4d706
Make the ReadDnsQueueAction tests retry on failures (#1114)
These tests are flaky due to some kind of contention/collision on the mock task
queue. Retrying seems to fix the vast majority of flakes, is easy to implement,
and is more performant than moving these tests into the fragileTests test suite.
2021-04-28 10:20:36 -04:00
gbrodman
d99278e723
Convert remaining read-only flow tests to dual-DB (#1107)
Note that there are many flow tests that aren't
@DualDatabaseTest-annotated yet but those will come later, as they will
require more changes to the flows (other PRs are coming or in progress).
This only includes the remaining EppResource flows that don't create a
history entry.
2021-04-27 20:37:09 -04:00
Ben McIlwain
9d4de806f5
Improve error when creating domain label lists for non-existent TLDs (#1112)
* Improve error message when creating domain label lists for non-existent TLDs
2021-04-27 19:17:23 -04:00
sarahcaseybot
2528ee05dd
Remove SMDRL completely from Datastore (#1104)
* Remove SMDRL completely from Datastore

* Remove some unnecessary stuff

* Change row count to 10000

* Remove implement EntityTestCase
2021-04-26 17:15:50 -04:00
Rachel Guan
367a38c5b0
Display changes when updating reserved list (#1093)
* add stageEntityChange to show diff

* add test cases
2021-04-26 13:31:57 -04:00
Lai Jiang
8884425a05
Fix build (#1109) 2021-04-26 10:34:29 -04:00
gbrodman
2c4c0bf9f8
Convert more tests to use @DualDatabaseTest and SQL in general (#1101)
Nothing super crazy here other than persisting the entity changes in
DomainDeleteFlow at the end of the flow rather than almost at the end.
This means that when we return the results we give the results as they
were originally present, rather than the subsequently-changed values.
2021-04-23 18:26:44 -04:00
Michael Muller
9c89643367
Fix Spec11 domain check (#1105)
* Fix Spec11 domain check

We should be checking to see if there are _any_ active domains for a given
reported domain, not to see if _the_ domain for the name is active.

The last change caused an exception for domains with soft-deleted past domains
of the same name.  The original code only checked the first domain returned
from the query, which may have been soft-deleted.  This version checks all
domain records to see if any are active.

* filter().count() -> anyMatch()
2021-04-23 14:20:31 -04:00
gbrodman
9f69a0bf2e
Begin saving the EppResource parent in *History objects (#1090)
* Begin saving the EppResource parent in *History objects

We use DomainCreateFlow as an example here of how this will work. There
were a few changes necessary:

- various changes around GracePeriod / GracePeriodHistory so that we can
actually store them without throwing NPEs
- Creating one injectable *History.Builder field and using in place of
the HistoryEntry.Builder injected field in DomainCreateFlow
- Saving the EppResource as the parent in the *History.Builder setParent
calls
- Converting to/from HistoryEntry/*History classes in
DatastoreTransactionManager. Basically, we'll want to return the
*History subclasses (and similar in the ofy portions of HistoryEntryDao)
- Converting a few HistoryEntry.Builder usages to DomainHistory.Builder
usages. Eventually we should convert all of them.
2021-04-22 15:03:37 -04:00
sarahcaseybot
40db04db8d
Use CommandWithRemoteApi in SetDatabaseTransitionScheduleCommand (#1099)
* Use CommandWithRemoteApi in ConfirmingCommand

* Remove unnecessary extensions

* Remove from ConfirmingCommand
2021-04-22 14:50:19 -04:00
Lai Jiang
217b37b9d5
Migrate the billing pipeline to flex template (#1100)
This is similar to the migration of the spec11 pipeline in #1073. Also removed
a few Dagger providers that are no longer needed.

TESTED=tested the dataflow job on alpha.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1100)
<!-- Reviewable:end -->
2021-04-22 10:26:15 -04:00
Lai Jiang
09b6e300fc
Remove unused BeamJpaExtension and related classes (#1102)
* Remove unused BeamJpaExtension and related classes

* Remove unused qualifiers
2021-04-22 10:02:18 -04:00
Lai Jiang
4d99a5dd35
Remove a linter warning (#1103)
* Remove a linter warning

* Remove duplicate
2021-04-22 09:42:05 -04:00
gbrodman
5d3e9da750
Defer all foreign keys in SQL (#1094)
* Defer all foreign keys in SQL

The main difference here is that the constraint violation exceptions
won't be thrown until the transaction is completed, rather than when the
insert is first performed within the transaction. We get the same error
message either way. The primary benefit to this is that when dealing
with large operations inside a single transaction (flows), we don't need
to worry about the order of insertions of removals with regards to
foreign keys.
2021-04-21 14:29:20 -04:00
Lai Jiang
464f9aed1f
Migrate Spec11 pipeline to flex template (#1073)
* Migrate Spec11 pipeline to flex template

Unfortunately this PR has turned out to be much bigger than I initially
conceived. However this is no good way to separate it out because the
changes are intertwined. This PR includes 3 main changes:

1. Change the spec11 pipline to use Dataflow Flex Template.
2. Retire the use of the old JPA layer that relies on credential saved
   in KMS.
3. Some extensive refactoring to streamline the logic and improve test
   isolation.

* Fix job name and remove projectId from options

* Add parameter logs

* Set RegistryEnvironment

* Remove logging and modify safe browsing API key regex

* Rename a test method and rebase

* Remove unused Junit extension

* Specify job region
2021-04-21 00:09:50 -04:00