Commit graph

3552 commits

Author SHA1 Message Date
Lai Jiang
d285edef3d
Fix a few linter warnings (#1122) 2021-05-04 13:35:31 -04:00
Weimin Yu
509c0dcd17
Handle bad production data when migrating to SQL (#1120)
* Handle bad production data when migrating to SQL

Ignore or fix bad entites when populating SQL with production data in
Datastore. These are mostly inconsistent foreign keys.

See b/185954992 for details.
2021-05-03 16:09:43 -04:00
sarahcaseybot
ce18bf0690
Use FakeClock to prevent Expired Certificate Violations (#1121)
* Use FakeClock to prevent Expired Certificate Violations

* Format fixes

* Make CertificateChecker static
2021-05-03 15:10:26 -04:00
Lai Jiang
8d63cbfca0
Remove enforcement date from the SslServerInitializer (#1117)
The enforcement data has passed and ICANN has confirmed that their web
WHOIS prober conforms to our requirements.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1117)
<!-- Reviewable:end -->
2021-04-30 15:44:03 -04:00
Lai Jiang
eb6a1fe1ed
Remove Pipeline as a field in pipeline classes (#1119)
In tests we use a TestPipelineExtension which does some static
initialization that should not be repeated the same JVM. In our
XXXPipeline classes we save the pipeline as a field and usually write lambdas
that are pass to the pipeline. Because lambdas are effectively anonymous inner
classes they are bound to their enclosing instances. When they get serialized
during pipeline execution, their enclosing classes also do. This might result
in undefined behavior when multiple lambdas in the same XXXPipeline are used
on the same JVM (such as in tests) where the static initialization may be done
multiple times if different class loaders are used. This is very
unlikely to happen but as a best practice we still remove them as
fields.
2021-04-30 14:32:33 -04:00
Weimin Yu
431710c95b
Improve usability of WipeOutCloudSqlAction (#1118)
* Improve usability of WipeOutCloudSqlAction

Replace the "drop owned" statement with ones that drops only tables and
sequences. The former statement also drops default grants for the
nomulus user, which must be restored before the database can be used by
the nomulus server and tools.
2021-04-29 23:09:20 -04:00
Michael Muller
1fdf9cb979
Convert GenerateLordnCommand to tm (#1091)
* Convert GenerateLordnCommand to tm

This makes use of QueryComposer and adds a `list()` method to it.

Since there was no test for GenerateLordnCommand, this also implements one.

* Changes requested in review

* Add test for list queries

* Stream domains instead of listing them

* Reformatted
2021-04-29 13:14:56 -04:00
Michael Muller
95fdd36c77
Make nom_build not check for ".git" directory (#1110)
* Make nom_build not check for ".git" directory

nom_build tries to verify that it is in the root of the tree prior to doing
anything, however checking for a .git directory doesn't work in a merged
directory.

* Minor formatting fix to attempt to force rebuild
2021-04-28 11:23:39 -04:00
Ben McIlwain
d239a4d706
Make the ReadDnsQueueAction tests retry on failures (#1114)
These tests are flaky due to some kind of contention/collision on the mock task
queue. Retrying seems to fix the vast majority of flakes, is easy to implement,
and is more performant than moving these tests into the fragileTests test suite.
2021-04-28 10:20:36 -04:00
gbrodman
d99278e723
Convert remaining read-only flow tests to dual-DB (#1107)
Note that there are many flow tests that aren't
@DualDatabaseTest-annotated yet but those will come later, as they will
require more changes to the flows (other PRs are coming or in progress).
This only includes the remaining EppResource flows that don't create a
history entry.
2021-04-27 20:37:09 -04:00
Ben McIlwain
9d4de806f5
Improve error when creating domain label lists for non-existent TLDs (#1112)
* Improve error message when creating domain label lists for non-existent TLDs
2021-04-27 19:17:23 -04:00
sarahcaseybot
2528ee05dd
Remove SMDRL completely from Datastore (#1104)
* Remove SMDRL completely from Datastore

* Remove some unnecessary stuff

* Change row count to 10000

* Remove implement EntityTestCase
2021-04-26 17:15:50 -04:00
Rachel Guan
367a38c5b0
Display changes when updating reserved list (#1093)
* add stageEntityChange to show diff

* add test cases
2021-04-26 13:31:57 -04:00
Lai Jiang
8884425a05
Fix build (#1109) 2021-04-26 10:34:29 -04:00
gbrodman
2c4c0bf9f8
Convert more tests to use @DualDatabaseTest and SQL in general (#1101)
Nothing super crazy here other than persisting the entity changes in
DomainDeleteFlow at the end of the flow rather than almost at the end.
This means that when we return the results we give the results as they
were originally present, rather than the subsequently-changed values.
2021-04-23 18:26:44 -04:00
Michael Muller
9c89643367
Fix Spec11 domain check (#1105)
* Fix Spec11 domain check

We should be checking to see if there are _any_ active domains for a given
reported domain, not to see if _the_ domain for the name is active.

The last change caused an exception for domains with soft-deleted past domains
of the same name.  The original code only checked the first domain returned
from the query, which may have been soft-deleted.  This version checks all
domain records to see if any are active.

* filter().count() -> anyMatch()
2021-04-23 14:20:31 -04:00
gbrodman
9f69a0bf2e
Begin saving the EppResource parent in *History objects (#1090)
* Begin saving the EppResource parent in *History objects

We use DomainCreateFlow as an example here of how this will work. There
were a few changes necessary:

- various changes around GracePeriod / GracePeriodHistory so that we can
actually store them without throwing NPEs
- Creating one injectable *History.Builder field and using in place of
the HistoryEntry.Builder injected field in DomainCreateFlow
- Saving the EppResource as the parent in the *History.Builder setParent
calls
- Converting to/from HistoryEntry/*History classes in
DatastoreTransactionManager. Basically, we'll want to return the
*History subclasses (and similar in the ofy portions of HistoryEntryDao)
- Converting a few HistoryEntry.Builder usages to DomainHistory.Builder
usages. Eventually we should convert all of them.
2021-04-22 15:03:37 -04:00
sarahcaseybot
40db04db8d
Use CommandWithRemoteApi in SetDatabaseTransitionScheduleCommand (#1099)
* Use CommandWithRemoteApi in ConfirmingCommand

* Remove unnecessary extensions

* Remove from ConfirmingCommand
2021-04-22 14:50:19 -04:00
Lai Jiang
217b37b9d5
Migrate the billing pipeline to flex template (#1100)
This is similar to the migration of the spec11 pipeline in #1073. Also removed
a few Dagger providers that are no longer needed.

TESTED=tested the dataflow job on alpha.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1100)
<!-- Reviewable:end -->
2021-04-22 10:26:15 -04:00
Lai Jiang
09b6e300fc
Remove unused BeamJpaExtension and related classes (#1102)
* Remove unused BeamJpaExtension and related classes

* Remove unused qualifiers
2021-04-22 10:02:18 -04:00
Lai Jiang
4d99a5dd35
Remove a linter warning (#1103)
* Remove a linter warning

* Remove duplicate
2021-04-22 09:42:05 -04:00
gbrodman
5d3e9da750
Defer all foreign keys in SQL (#1094)
* Defer all foreign keys in SQL

The main difference here is that the constraint violation exceptions
won't be thrown until the transaction is completed, rather than when the
insert is first performed within the transaction. We get the same error
message either way. The primary benefit to this is that when dealing
with large operations inside a single transaction (flows), we don't need
to worry about the order of insertions of removals with regards to
foreign keys.
2021-04-21 14:29:20 -04:00
Lai Jiang
464f9aed1f
Migrate Spec11 pipeline to flex template (#1073)
* Migrate Spec11 pipeline to flex template

Unfortunately this PR has turned out to be much bigger than I initially
conceived. However this is no good way to separate it out because the
changes are intertwined. This PR includes 3 main changes:

1. Change the spec11 pipline to use Dataflow Flex Template.
2. Retire the use of the old JPA layer that relies on credential saved
   in KMS.
3. Some extensive refactoring to streamline the logic and improve test
   isolation.

* Fix job name and remove projectId from options

* Add parameter logs

* Set RegistryEnvironment

* Remove logging and modify safe browsing API key regex

* Rename a test method and rebase

* Remove unused Junit extension

* Specify job region
2021-04-21 00:09:50 -04:00
sarahcaseybot
a0995fa0eb
Stop dual read and dual write of SMDRL (#1095)
* Stop dual read and dual write of SMDRL

* Remove some more stuff from SignedMarkRevocationListDaoTest

* Change some names
2021-04-20 17:08:59 -04:00
Weimin Yu
fff95b20e6
Skip undefined secrets in keyring migration (#1098)
* Skip undefined secrets in keyring migration

If a secret does not exist in datastore, log and skip it.
2021-04-20 16:26:40 -04:00
gbrodman
23896b64c7
Set default value of 1 for new not-null columns (#1097)
Use 1 since it's the constant singleton ID
2021-04-20 15:25:20 -04:00
Ben McIlwain
844b5ab713
Send an immediate poll message for superuser domain deletes (#1096)
* Send an immediate poll message for superuser domain deletes

This poll message is in addition to the normal poll message that is sent when
the domain's deletion is effective (typically 35 days later). It's needed
because, in the event of a superuser deletion, the owning registrar won't
otherwise necessarily know it's happening.

Note that, in the case of a --immediate superuser deletion, the normal poll
message is already being sent immediately, so this additional poll message is
not necessary.
2021-04-20 15:22:49 -04:00
sarahcaseybot
aac952d6a3
Return to using hash for login validation (#1084)
* Return to using hash for login validation

This PR also removes the start date for certificate enforcement.

* Inline verify certificate compliance
2021-04-20 14:07:01 -04:00
gbrodman
ee31f1fd95
Update various tests to work with SQL as well (#1078)
* Update various tests to work with SQL as well

The main weird bit here is adding a method in DatabaseHelper to
retrieve and initialize all objects in either database. The
initialization is necessary since it's used post-command-dry-run to make
sure that no changes were actually made.
2021-04-20 11:52:53 -04:00
Michael Muller
4657be21b7
Convert CountDomainsCommand to tm (#1092)
* Convert CountDomainsCommand to tm

As part of this, implement "select count(*)" queries in the QueryComposer.

* Replaced kludgy trick for objectify count
2021-04-20 10:38:38 -04:00
sarahcaseybot
48732c51e8
Always use Cloud SQL as primary in SignedMarkRevocationListDao (#1061)
* Modify ClaimsList DAO to always use Cloud SQL as primary

* Revert ClaimsList add changes to SignedMarkRevocationList

* Fix flow tests

* Use start of time for empty list

* replace lambda with method reference
2021-04-19 14:51:00 -04:00
Weimin Yu
7893ba746a
Upload latest version of RDE report to icann (#1089)
* Upload latest version of RDE report to icann

Currently the RdeReportAction is hard coded to load the initial version
of a report. This is wrong when reports have been regenerated.

Changed lines are copied from RdeUploadAction.
2021-04-16 17:12:02 -04:00
Michael Muller
1c96cd64fe
Implement query abstraction (#1069)
* Implement query abstraction

Implement a query abstraction layer ("QueryComposer") that allows us to
construct fluent-style queries that work across both Objectify and JPA.

As a demonstration of the concept, convert Spec11EmailUtils and its test to
use the new API.

Limitations:
-  The primary limitations of this system are imposed by datastore, for
   example all queryable fields must be indexed, orderBy must coincide with
   the order of any inequality queries, inequality filters are limited to one
   property...
-  JPA queries are limited to a set of where clauses (all of which must match)
   and an "order by" clause.  Joins, functions, complex where logic and
   multi-table queries are simply not allowed.
-  Descending sort order is currently unsupported (this is simple enough to
   add).
2021-04-16 12:21:03 -04:00
Ben McIlwain
bc2a5dbc02
Fix bug that was incorrectly assuming Cursor would always exist (#1088)
* Fix bug that was incorrectly assuming Cursor would always exist

In fact, the Cursor entity does not always exist (i.e. if an upload has never
previously been done on this TLD, i.e. it's a new TLD), and the code needs to be
resilient to its non-existence.

This bug was introduced in #1044.
2021-04-15 17:03:25 -04:00
Weimin Yu
98d259449b
Use lazy injection in SendEscrow command (#1086)
* Use lazy injection in SendEscrow command

The injected object in SendEscrowReportToIcannCommand creates Ofy keys
in its static initialization routine. This happens before the RemoteApi
setup. Use lazy injection to prevent failure.
2021-04-15 16:15:01 -04:00
gbrodman
1cc8af4acd
Specify explicit ofyTm usage in SetDatabaseTransitionScheduleCommand (#1081)
* Specify explicit ofyTm usage in SetDatabaseTransitionScheduleCommand

We cannot use the standard MutatingCommand because the DB schedule is
explicitly always stored in Datastore, and once we transition to
SQL-as-primary, MutatingCommand will stage the entity changes to SQL.

In addition, we remove the raw ofy() call from the test.
2021-04-15 11:59:04 -04:00
Rachel Guan
fbef643488
make transitionId a required parameter (#1083) 2021-04-15 10:42:15 -04:00
Lai Jiang
2161e46a4b
Fix a typo (#1085) 2021-04-15 08:15:31 -04:00
Lai Jiang
d7f27bdad3
Update the gradle appengine plugin (#1082) 2021-04-14 19:33:55 -04:00
sarahcaseybot
78e139b2c8
Add a ComparePremiumLists command (#1056)
* Add a ComparePremiumLists command

* Add a command description

* fix output

* Fix comment format

* Add periods

* Small output message change

* Inline getting stdout

* Use sets

* Inline Sets.difference
2021-04-14 18:10:47 -04:00
gbrodman
87d511d5e3
Convert more classes to using SQL / TM (#1067)
* Convert more classes to using SQL / TM

Nothing much particularly crazy here
2021-04-14 16:45:06 -04:00
sarahcaseybot
eff79e9c99
Remove unecessary ClaimsList in FlowTest (#1077) 2021-04-14 13:49:35 -04:00
Weimin Yu
bb453b1982
Migrate Keyring secrets to Secret Manager (#1072)
* Migrate Keyring secrets to Secret Manager

Implented dual-read of Keyring secrets with Datastore as primary.

Implemented dual-write of keyring secrets with Datastore as primary.
Secret manager write failures are simply thrown. This is fine since all
keyring writes are manual, throught eh update_kms_keyring command.

Added a one-way migration command that copies all data to secret manager
(unencrypted).
2021-04-14 10:17:33 -04:00
Weimin Yu
8b41b5c76f
Upgrade testcontainers to work around a race (#1080)
* Upgrade testcontainers to work around a race

testcontainers 1.15.? has a race condition that occassionally causes deadlocks.
This can be worked around by upgrading to 1.15.2 and set transport type to
http5.

See https://github.com/testcontainers/testcontainers-java/issues/3531
for more information.

There are two changes that are not lockfiles:
- dependencies.gradle
- java_common.gradle
2021-04-14 09:45:09 -04:00
Lai Jiang
881f0f5f09
Make cross referencing work in Kythe, take 2 (#1079)
* Make cross referencing work in Kythe, take 2

Per suggestions on b/184284124.
2021-04-14 09:13:05 -04:00
Weimin Yu
abe6a193a8
Add hoc tool to fix duplicate contactId (#1076)
* Add hoc tool to fix duplicate contactId
2021-04-13 22:29:22 -04:00
gbrodman
d35460f14c
Convert TmchCrl and ServerSecret to cleaner tm() impls (#1068)
* Convert TmchCrl and ServerSecret to cleaner tm() impls

When I implemented this originally I knew a lot less than I know now
about how we'll be storing and retrieving these singletons from SQL. The
optimal way here is to use the single SINGLETON_ID as the primary key,
that way we always know how to create the key that we can use in the
tm() retrieval.

This allows us to use generic tm() methods and to remove the handcrafted
SQL queries.
2021-04-13 20:50:07 -04:00
gbrodman
245e2ea5a8
Enforce consistency in non-cached FKI loads (#1075)
* Enforce consistency in non-cached FKI loads

For the cached code path, we do not require consistency but we do
require the ability to load / operate on large numbers of entities (so,
we must do so without a Datastore transaction). For the non-cached code
path, we require consistency but do not care about large numbers of
entities, so we must remain in the transaction that we're already in.
2021-04-13 15:14:02 -04:00
sarahcaseybot
65f35ac8c1
Fix TimestampInversionException (#1065)
* Fix TimestampInversionException

* Add autoIncrement

* unset auto increment mode
2021-04-13 11:59:14 -04:00
sarahcaseybot
994af085d8
Add a CompareReservedListCommand (#1054)
* Add a CompareReservedListCommand

* compare maps

* output format fixes

* Clean up loops

* Inline Sets.difference()

* Remove ImmutableCopy()
2021-04-13 11:45:45 -04:00