Commit graph

3790 commits

Author SHA1 Message Date
Ben McIlwain
6ffe84e93d
Add a scrap command to hard-delete a host resource (#1391) 2021-10-15 12:28:18 -04:00
Ben McIlwain
a451524010
Add tests for obscure hostname canonicalization rule (#1388)
Also correctly configures Gradle for the util subproject (it wasn't possible to
run tests in IntelliJ without these changes).
2021-10-14 14:53:28 -04:00
Rachel Guan
bb8988ee4e
Set payload in success response after sending notification emails (#1377)
* Set payload in success response after sending expiring certificate notification emails

* Modify log message and test cases for run() in sendExpiringCertificateNotificationEmailAction
2021-10-13 15:58:25 -04:00
Rachel Guan
2aff72b3b6
Add reason and requestedByRegistrar to domain renew flow (#1378)
* Resolve merge conflict

* Include reason and requestedByRegistrar in URS test file

* Modify test cases for new parameters in renew flow

* Add reason and registrar_request to renew domain command

* Update comments for new params in renew flow

* Make changes based on feedback
2021-10-13 11:41:02 -04:00
Weimin Yu
35fd61f771
Update parameter to Datastore wipe pipeline (#1385)
* Update parameter to Datastore wipe pipeline

Add the newly required RegistryEnvironment parameter to
BulkDeleteDatastorePipeline.

Remove the nullable annotation for this parameter in options
class.

Update metadata files regarding this parameter.
2021-10-11 17:31:50 -04:00
Michael Muller
13cb17e9a4
Implement several fixes affecting test flakiness (#1379)
* Implement several fixes affecting test flakiness

- Continued to do transaction manager cleanups on reply failure (lack of this
  may be causing cascading failures.
- Fix UpdateDomainCommandTest's output check (the test was checking for error
  output in standard error, but the command writes its output to the logs.
  Apparently, these may or may not be reflected in standard error depending on
  current global state)
- Remove unnecessary locking and incorrect comment in CommandTestCase.  The
  JUnit tests are not run in parallel in the same JVM and, in general, there
  are much bigger obstacles to this than standard output stream locking.

* Fix bad log message check
2021-10-11 12:54:03 -04:00
Ben McIlwain
4f1c317bbc
Revert update auto timestamp non-transactional fallback (#1380)
This was added recently in PR #1341 as an attempted fix for our test flakiness,
but it turns out that it didn't address the root issue (whereas PR #1361
did). So this removes the fallback, as there's no reason this should ever be
called outside of a transactional context.
2021-10-08 16:44:45 -04:00
gbrodman
c8aa32ef05
Include more info in host/domain name failures (#1346)
We're seeing some of these in CreateSyntheticHistoryEntriesAction and I
can't tell why from the logs (it doesn't appear to print the repo ID or
domain/host name)
2021-10-08 15:17:22 -04:00
gbrodman
95a1bbf66a
Temporarily disable SQL->DS replay in all tests (#1363) 2021-10-08 14:15:57 -04:00
Rachel Guan
23aa16469e
Add WipeOutContactHistoryPiiAction to prod (#1356) 2021-10-08 11:46:26 -04:00
Ben McIlwain
0277c5c25a
Add TmOverrideExtension for more safe TM overrides in tests (#1382)
* Add TmOverrideExtension for more safe TM overrides in tests

This is safer to use than calling setTmForTest() directly because this extension
also handles the corresponding call to removeTmOverrideForTest() automatically,
the forgetting of which has been a source of test flakiness/instability in the
past.

There are now broadly two ways to get tests to run in JPA: either use
DualDatabaseTest, an AppEngineExtension, and the corresponding JPA-specific
@Test annotations, OR use this override alongside a
JpaTransactionManagerExtension.
2021-10-07 19:26:25 -04:00
Ben McIlwain
b1b0589281
Elaborate on database read-only error message (#1355)
* Elaborate on database read-only error message
2021-10-07 13:25:24 -04:00
Ben McIlwain
28628564cc
Set response payload when wiping out contact history PII (#1376)
Also uses smaller batches in tests so that they don't take so long.
2021-10-07 12:43:41 -04:00
Michael Muller
835f93f555
Add a reference to RDAP conformance checker (#1358)
* Add a reference to RDAP conformance checker

Make a note of the RDAP conformance checker for the next time that we deal
with the RDAP code - would be nice to have this in the test suite.

* Reformat comment
2021-10-07 12:34:41 -04:00
Ben McIlwain
276c188e9d
Canonicalize domain/host names in initial import script (#1347)
* Canonicalize domain/host names in initial import script

* Add tests and make reduce some method visibility
2021-10-07 11:59:46 -04:00
Rachel Guan
34ecc6fbe7
Add new parameter renew_one_year to URS (#1364)
* Add autorenews to URS (#1343)

* Add autorenews to URS

* Add autorenews to existing xml files for test cases

* Harmonize domain.get() in existing code

* Fix typo in test case name

* Modify existing test helper method to allow testing with different domain bases
2021-10-06 20:40:43 -04:00
gbrodman
0f4156c563
Use a more efficient query to find resources in histories (#1354) 2021-10-06 15:20:31 -04:00
Michael Muller
e1827ab939
Defer python discovery until presubmit task (#1352)
* Customize LGTM build command

Our presubmit requires a version of python that is more recent than what
lgtm.com's build environments have installed.  Instead of trying to upgrade
them or downgrade our python version, just do the steps of the build that LGTM
needs (i.e. just build the main classes and test classes).
2021-10-06 10:09:13 -04:00
Ben McIlwain
51b2887709
Fix BigQuery data set name handling in activity reporting (#1361)
* Fix BigQuery data set name handling in activity reporting

This is not a constant (as it depends on runtime state), so it can't be named
using UPPER_SNAKE_CASE. Additionally, it's not good practice to use field
initialization when there's logic depending on runtime state involved. So this
PR changes the class to use constructor injection and moves the logic into the
constructor.

* Add fix for ICANN reporting provide

* Extract out ICANN reporting data set

* Inject TransactionManager

* Make TransactionInfo static (per Mike)

* Use ofyTm() in BackupTestStore

* Revert extraneous formatting

* Use auditedOfy in CommitLogMutationTest
2021-10-05 15:11:03 -04:00
Lai Jiang
62eb8801c5
Finish RDE pipeline implementation in SQL mode (#1330)
This PR adds the final step in RDE pipeline (enqueueing the next action
  to Cloud Tasks) and makes some necessary changes, namely by making all
  CloudTasksUtils related classes serializable, so that they can be used
  on Beam.

<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1330)
<!-- Reviewable:end -->
2021-10-04 21:02:44 -04:00
Lai Jiang
f6920454f6
Fix the beam staging script, take 3 (#1370)
The number of arguments changed in https://github.com/google/nomulus/pull/1369, so the check needs to change as well.

<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1370)
<!-- Reviewable:end -->
2021-10-04 16:44:32 -04:00
Lai Jiang
9103216a46
Fix beam deployment script again. (#1369)
uberjar task and uberjar name are now different (beamPipelineCommon and
beam_pipeline_common, respectively). This is more idiomatic with regard
to naming conventions but we need to take two different variables now.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1369)
<!-- Reviewable:end -->
2021-10-04 14:23:28 -04:00
Lai Jiang
c6705d1956
Fix sandbox cron (#1366)
* Fix sandbox cron

"synchronized" can only be used to specify a 24h time range that is
evenly divided by the interval value, e. g. "every 2 hours
synchronized".

* Change to a different time
2021-10-04 11:09:55 -04:00
Lai Jiang
737f65bd33
Change Beam uber jar name in Nomulu release GCB config (#1367)
The uber jar name was changed in #1351.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1367)
<!-- Reviewable:end -->
2021-10-04 10:47:27 -04:00
Lai Jiang
c8caa8f80b
Remove the use of AppEngineEnvironment in Spec11Pipeline (#1365)
After #1348 it is no longer necessary to use AppEngineEnvironment in
Beam pipelines. In tests it is taken care of by the
DatastoreEntityExtension whereas on Dataflow the
RegistryPipelineWorkerInitializer does the same initialization for Ofy.
2021-10-02 19:23:09 -04:00
Rachel Guan
65ef18052b
Add autorenews to URS (#1343)
* Add autorenews to URS

* Add autorenews to existing xml files for test cases
2021-10-01 19:11:46 -04:00
Lai Jiang
f7938e80f7
Streamline how to fake an App Engine environment (#1348)
Both `DatastoreEntityExtension.PlaceholderEnvironment` and `AppEngineEnvironment` does the same thing, so there is no point having both of them exist. To use `AppEngineEnvionrment` as an autoclosable requires the user to be mindful of where a fake App Engine environment is required. It is better to set this either in the `DatastoreEntityExtension` for tests, or in the worker initializer in Beam. It also makes it easier to remove the fake environment when we are completely datastore free.

Also made a change to how `IdService` allocate Ids in Beam.
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1348)
<!-- Reviewable:end -->
2021-10-01 16:46:46 -04:00
Lai Jiang
d8b3a30a20
Rename UpdateKmsKeyringCommand (#1353)
This brings it in line with GetKeyringSecretCommand. We still need to
remove the rest of remaining Cloud KMS related code in the future.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1353)
<!-- Reviewable:end -->
2021-10-01 16:45:45 -04:00
sarahcaseybot
93715c6f9e
Add VKey workaround to spec11 pipeline (#1339)
* Add VKey workaround to spec11 pipeline

* Parallelize entity loading
2021-10-01 15:21:16 -04:00
Rachel Guan
90cf4519c5
Add a cron job to periodically empty out fields on deleted entities t… (#1303)
* Add a cron job to periodically empty out fields on deleted entities that are at least 18 months old

* Process ContactHistory entities via batching

* Improve test cases by not making assertions in a loop
2021-09-30 15:17:37 -04:00
Michael Muller
3a177f36b1
Make :core:cleanTest depend on FilterTests (#1342)
* Make :core:cleanTest depend on FilterTests

The "cleanTest" target doesn't work for our specialized tests derived from
FilterTest.  Make them all explicit dependencies of cleanTest so we can reset
the tests from a single target.
2021-09-30 10:46:36 -04:00
Lai Jiang
fbbe014e96
Make it possible to stage a single Beam pipeline (#1351) 2021-09-29 18:27:23 -04:00
Ben McIlwain
b05b77cfd1
Add/use more DatabaseHelper convenience methods (#1327)
* Add/use more DatabaseHelper convenience methods

This also fixes up some existing uses of "put" in test code that should be
inserts or updates (depending on which is intended). Doing an insert/update
makes stronger guarantees about an entity either not existing or existing,
depending on what you're doing.

* Convert more Object -> ImmutableObject

* Merge branch 'master' into tx-manager-sigs

* Revert breaking PremiumListDao change

* Refactor more insertInDb()

* Fight more testing errors

* Merge branch 'master' into tx-manager-sigs

* Merge branch 'master' into tx-manager-sigs

* Merge branch 'master' into tx-manager-sigs

* Merge branch 'master' into tx-manager-sigs

* Add removeTmOverrideForTest() calls

* Merge branch 'master' into tx-manager-sigs
2021-09-28 17:16:54 -04:00
Michael Muller
420a0b8b9a
Use debian10 image for builder, not ubuntu1804 (#1345)
The debian10 image is generally a bit more recent and, in particular, includes
python 3.7.3, which we're currently using as a baseline for our builds.
2021-09-28 14:49:13 -04:00
gbrodman
cc062e3528
Reduce # shards in CreateSyntheticHistoryEntriesAction (#1344)
We need these to get created (we are blocked from moving to SQL until 30
days after their creation) so reduce this to 3 in the hopes of avoiding
the SQL overloads while we debug why those are occurring in the first
place.
2021-09-28 10:46:50 -04:00
Michael Muller
56a0e35314
Find a suitable version of python. (#1338)
* Find a suitable version of python.

When running presubmit, we were using /usr/bin/python3, which works fine on
systems that have a reasonably recent python version there.  However, our CI
system has a very old version of python there and prefers the use of "pyenv"
to modify the PATH to provide the desired version of python as simply
"python".  So add a check to use the first of "python" or "/usr/bin/python3"
that is at least version 3.7.3.
2021-09-27 16:43:45 -04:00
sarahcaseybot
de434f861f
Migrate ICANN activity reports to Cloud SQL on BQ (#1332)
* Migrate ICANN activity reports to Cloud SQL on BQ

* Fix data set name
2021-09-27 15:27:20 -04:00
Ben McIlwain
3caee5fba7
Improve some log messages for readability/consistency (#1333)
* Improve some log messages for readability/consistency

* Address code review comments
2021-09-27 11:35:14 -04:00
Ben McIlwain
ff3c848def
Add handling for UpdateAutoTimestamp when not in a transaction (#1341)
* Add handling for UpdateAutoTimestamp when not in a transaction

It's not clear why this is sometimes causing test flakes, but getting better
logging involved should help clear it up.

This also changes AppEngineExtension to insert without reloading the initial
test data, rather than putting it (potentially involving a merge) and reloading
it in a separate transaction. This should hopefully reduce the chance of weird
conflicts.
2021-09-27 11:32:15 -04:00
Rachel Guan
f0b3be5bb6
Improves test file for SendExpiringCertificateNotificationEmailAction (#1335)
* Improves test cases for SendExpiringCertificateNotificationEmailAction
2021-09-27 09:56:16 -04:00
gbrodman
18b808bd34
Fix injection with BackfillRegistryLocksCommand (#1337)
It would have been nice if this had failed at compile-time rather than
an NPE, but we need to make sure to specify that we need to inject this
command to get e.g. the random string generator

In addition, print out only the names of the failed domains (rather than
the entire domain object) for readability.
2021-09-24 14:08:30 -04:00
Lai Jiang
d7689539d7
Remove mention of bazel run (#1340)
Also provides a workaround in the error message.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1340)
<!-- Reviewable:end -->
2021-09-24 11:44:27 -04:00
Lai Jiang
c14ce6866b
Remove remnants of JUnit 4 rules (#1336)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1336)
<!-- Reviewable:end -->
2021-09-24 06:35:35 -04:00
Michael Muller
3b84542e46
Add a presubmit to verify no new JS dependencies (#1334)
* Add a presubmit to verify no new JS dependencies

Verify that we have a known set of javascript dependencies.  This guards
against the inadvertent introduction of a new dependency with a disallowed
license.

TESTED: Added a new package to packages.json, observed presubmit failure.

* Replaced f-strings, printed python version

For some reason, it looks like we're using a python version older than 3.6 on
our CI machines.

* Remove python version trace.
2021-09-23 14:42:47 -04:00
Lai Jiang
fc7db91d70
Consolidate the use of URL parameters to specify database override (#1331)
There are actions for which we want to provide an override for the database
to use, like when launching Spec11 and Invoicing pipelines. It make sense to
consolidate around the same parameter provided from the same module for
consistency in all cases, instead of defining an override for each action.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1331)
<!-- Reviewable:end -->
2021-09-22 20:01:19 -04:00
Weimin Yu
3d8aa85d63
Fix ReadOnlyCheckingQuery's streaming method (#1329)
* Fix ReadOnlyCheckingQuery's streaming method

Following up to PR 1314: fix one more query defaulting to List when
stream() is invoked.
2021-09-21 15:40:50 -04:00
gbrodman
e14cd8bfa2
Add locking and a response in ReplicateToDatastoreAction (#1328)
* Add locking and a response in ReplicateToDatastoreAction

The response is necessary to get nicer logs in GAE and nicer cron job
behavior.

In addition:
- fix issues where locks would be backed up and replayed to Datastore
(they shouldn't be replayed)
- do ignore-read-only writes when replaying the transactions
2021-09-21 10:12:27 -04:00
Michael Muller
82e8641816
Fix javadoc problems with SoyInfo and subprojects (#1326)
* Fix javadoc problems with SoyInfo and subprojects

The *SoyInfo.java files generated by the soy compiler contain deprecation
warnings with links to files that are not imported.  This causes a javadoc
warning.  Temporarily fix this by replacing the link tags with "LINK".  This
also allows us to remove the exclusion of these files, which is a bit nicer.

Also disable javadoc tasks from subprojects.  These just break because they
don't have access to the legacy javadoc classes in the root.
2021-09-20 10:35:07 -04:00
gbrodman
7461ada0fb
Include Cursor in the initial SQL population (#1323)
The test for this also required a bit of a fix in the Cursor scope
initialization. If you persist a Key<?> in some object in Datastore, it
persists just the standard data you'd expect, basically the parent, the
kind, and the object's ID/name. Then, when you load it back in from
Datastore it uses the app ID of whatever environment you're loading in
(the Key contains this info even though it isn't included in the
toString() of Key)

If you persist the websafe string format of a Key (which is what we do
for Cursors), it includes the app ID so when you load it, it contains
the old app ID not the new one (if the app ID has changed).

In the pipelines, we use the standard default environment which has a
different app ID from the test environment that's set up by the
DatastoreEntityExtension.

As a result, we should check in Cursor to see if the key is *any*
cross-tld-key, rather than the exact one that exists within this app
environment.
2021-09-19 09:23:02 -04:00
Ben McIlwain
d91ca0eb8a
Clean up tx manager insert() signature and add convenience helper method (#1325)
* Clean up tx manager insert() signature and add convenience helper method

This is the first of a series of PRs to clean up the type signatures on the
TransactionManager methods (which are way too generic), along with creating some
helper methods for use in tests only that don't require creating transactions
all over the place, thus reducing visual noise at callsites. This first method
is DatabaseHelper.insertInDb(), but there will be plenty of others. Note that
this is only for the Cloud SQL transaction manager -- I'm not bothering to
migrate any Datastore-only code, as that will be going away soon enough.
2021-09-17 14:45:07 -04:00