Commit graph

3629 commits

Author SHA1 Message Date
sarahcaseybot
a0995fa0eb
Stop dual read and dual write of SMDRL (#1095)
* Stop dual read and dual write of SMDRL

* Remove some more stuff from SignedMarkRevocationListDaoTest

* Change some names
2021-04-20 17:08:59 -04:00
Weimin Yu
fff95b20e6
Skip undefined secrets in keyring migration (#1098)
* Skip undefined secrets in keyring migration

If a secret does not exist in datastore, log and skip it.
2021-04-20 16:26:40 -04:00
gbrodman
23896b64c7
Set default value of 1 for new not-null columns (#1097)
Use 1 since it's the constant singleton ID
2021-04-20 15:25:20 -04:00
Ben McIlwain
844b5ab713
Send an immediate poll message for superuser domain deletes (#1096)
* Send an immediate poll message for superuser domain deletes

This poll message is in addition to the normal poll message that is sent when
the domain's deletion is effective (typically 35 days later). It's needed
because, in the event of a superuser deletion, the owning registrar won't
otherwise necessarily know it's happening.

Note that, in the case of a --immediate superuser deletion, the normal poll
message is already being sent immediately, so this additional poll message is
not necessary.
2021-04-20 15:22:49 -04:00
sarahcaseybot
aac952d6a3
Return to using hash for login validation (#1084)
* Return to using hash for login validation

This PR also removes the start date for certificate enforcement.

* Inline verify certificate compliance
2021-04-20 14:07:01 -04:00
gbrodman
ee31f1fd95
Update various tests to work with SQL as well (#1078)
* Update various tests to work with SQL as well

The main weird bit here is adding a method in DatabaseHelper to
retrieve and initialize all objects in either database. The
initialization is necessary since it's used post-command-dry-run to make
sure that no changes were actually made.
2021-04-20 11:52:53 -04:00
Michael Muller
4657be21b7
Convert CountDomainsCommand to tm (#1092)
* Convert CountDomainsCommand to tm

As part of this, implement "select count(*)" queries in the QueryComposer.

* Replaced kludgy trick for objectify count
2021-04-20 10:38:38 -04:00
sarahcaseybot
48732c51e8
Always use Cloud SQL as primary in SignedMarkRevocationListDao (#1061)
* Modify ClaimsList DAO to always use Cloud SQL as primary

* Revert ClaimsList add changes to SignedMarkRevocationList

* Fix flow tests

* Use start of time for empty list

* replace lambda with method reference
2021-04-19 14:51:00 -04:00
Weimin Yu
7893ba746a
Upload latest version of RDE report to icann (#1089)
* Upload latest version of RDE report to icann

Currently the RdeReportAction is hard coded to load the initial version
of a report. This is wrong when reports have been regenerated.

Changed lines are copied from RdeUploadAction.
2021-04-16 17:12:02 -04:00
Michael Muller
1c96cd64fe
Implement query abstraction (#1069)
* Implement query abstraction

Implement a query abstraction layer ("QueryComposer") that allows us to
construct fluent-style queries that work across both Objectify and JPA.

As a demonstration of the concept, convert Spec11EmailUtils and its test to
use the new API.

Limitations:
-  The primary limitations of this system are imposed by datastore, for
   example all queryable fields must be indexed, orderBy must coincide with
   the order of any inequality queries, inequality filters are limited to one
   property...
-  JPA queries are limited to a set of where clauses (all of which must match)
   and an "order by" clause.  Joins, functions, complex where logic and
   multi-table queries are simply not allowed.
-  Descending sort order is currently unsupported (this is simple enough to
   add).
2021-04-16 12:21:03 -04:00
Ben McIlwain
bc2a5dbc02
Fix bug that was incorrectly assuming Cursor would always exist (#1088)
* Fix bug that was incorrectly assuming Cursor would always exist

In fact, the Cursor entity does not always exist (i.e. if an upload has never
previously been done on this TLD, i.e. it's a new TLD), and the code needs to be
resilient to its non-existence.

This bug was introduced in #1044.
2021-04-15 17:03:25 -04:00
Weimin Yu
98d259449b
Use lazy injection in SendEscrow command (#1086)
* Use lazy injection in SendEscrow command

The injected object in SendEscrowReportToIcannCommand creates Ofy keys
in its static initialization routine. This happens before the RemoteApi
setup. Use lazy injection to prevent failure.
2021-04-15 16:15:01 -04:00
gbrodman
1cc8af4acd
Specify explicit ofyTm usage in SetDatabaseTransitionScheduleCommand (#1081)
* Specify explicit ofyTm usage in SetDatabaseTransitionScheduleCommand

We cannot use the standard MutatingCommand because the DB schedule is
explicitly always stored in Datastore, and once we transition to
SQL-as-primary, MutatingCommand will stage the entity changes to SQL.

In addition, we remove the raw ofy() call from the test.
2021-04-15 11:59:04 -04:00
Rachel Guan
fbef643488
make transitionId a required parameter (#1083) 2021-04-15 10:42:15 -04:00
Lai Jiang
2161e46a4b
Fix a typo (#1085) 2021-04-15 08:15:31 -04:00
Lai Jiang
d7f27bdad3
Update the gradle appengine plugin (#1082) 2021-04-14 19:33:55 -04:00
sarahcaseybot
78e139b2c8
Add a ComparePremiumLists command (#1056)
* Add a ComparePremiumLists command

* Add a command description

* fix output

* Fix comment format

* Add periods

* Small output message change

* Inline getting stdout

* Use sets

* Inline Sets.difference
2021-04-14 18:10:47 -04:00
gbrodman
87d511d5e3
Convert more classes to using SQL / TM (#1067)
* Convert more classes to using SQL / TM

Nothing much particularly crazy here
2021-04-14 16:45:06 -04:00
sarahcaseybot
eff79e9c99
Remove unecessary ClaimsList in FlowTest (#1077) 2021-04-14 13:49:35 -04:00
Weimin Yu
bb453b1982
Migrate Keyring secrets to Secret Manager (#1072)
* Migrate Keyring secrets to Secret Manager

Implented dual-read of Keyring secrets with Datastore as primary.

Implemented dual-write of keyring secrets with Datastore as primary.
Secret manager write failures are simply thrown. This is fine since all
keyring writes are manual, throught eh update_kms_keyring command.

Added a one-way migration command that copies all data to secret manager
(unencrypted).
2021-04-14 10:17:33 -04:00
Weimin Yu
8b41b5c76f
Upgrade testcontainers to work around a race (#1080)
* Upgrade testcontainers to work around a race

testcontainers 1.15.? has a race condition that occassionally causes deadlocks.
This can be worked around by upgrading to 1.15.2 and set transport type to
http5.

See https://github.com/testcontainers/testcontainers-java/issues/3531
for more information.

There are two changes that are not lockfiles:
- dependencies.gradle
- java_common.gradle
2021-04-14 09:45:09 -04:00
Lai Jiang
881f0f5f09
Make cross referencing work in Kythe, take 2 (#1079)
* Make cross referencing work in Kythe, take 2

Per suggestions on b/184284124.
2021-04-14 09:13:05 -04:00
Weimin Yu
abe6a193a8
Add hoc tool to fix duplicate contactId (#1076)
* Add hoc tool to fix duplicate contactId
2021-04-13 22:29:22 -04:00
gbrodman
d35460f14c
Convert TmchCrl and ServerSecret to cleaner tm() impls (#1068)
* Convert TmchCrl and ServerSecret to cleaner tm() impls

When I implemented this originally I knew a lot less than I know now
about how we'll be storing and retrieving these singletons from SQL. The
optimal way here is to use the single SINGLETON_ID as the primary key,
that way we always know how to create the key that we can use in the
tm() retrieval.

This allows us to use generic tm() methods and to remove the handcrafted
SQL queries.
2021-04-13 20:50:07 -04:00
gbrodman
245e2ea5a8
Enforce consistency in non-cached FKI loads (#1075)
* Enforce consistency in non-cached FKI loads

For the cached code path, we do not require consistency but we do
require the ability to load / operate on large numbers of entities (so,
we must do so without a Datastore transaction). For the non-cached code
path, we require consistency but do not care about large numbers of
entities, so we must remain in the transaction that we're already in.
2021-04-13 15:14:02 -04:00
sarahcaseybot
65f35ac8c1
Fix TimestampInversionException (#1065)
* Fix TimestampInversionException

* Add autoIncrement

* unset auto increment mode
2021-04-13 11:59:14 -04:00
sarahcaseybot
994af085d8
Add a CompareReservedListCommand (#1054)
* Add a CompareReservedListCommand

* compare maps

* output format fixes

* Clean up loops

* Inline Sets.difference()

* Remove ImmutableCopy()
2021-04-13 11:45:45 -04:00
Lai Jiang
ce25cea134
Disable TLS tests related to v1.1 (#1074)
There is no need for this test now because we've past the enforcement
date. We should take out the entire enforcement date logic but right now
this test is failing because TLS 1.1 is not being supported anymore by
the latest release of JDK 11.

The other test is a bit tricky to fix, see comment.

Disable these tests for now to unblock development.
2021-04-13 10:30:58 -04:00
gbrodman
92dcacf78c
Add a beforeSqlSave callback to ReplaySpecializer (#1062)
* Add a beforeSqlSave callback to ReplaySpecializer

When in the Datastore-primary and SQL-secondary stage, we will want to
save the EppResource-at-this-point-in-time field in the *History
objects so that later on we can examine the *History objects to see what
the resource looked like at that point in time.

Without this PR, the full object at that point in time would be lost
during the asynchronous replay since Datastore doesn't know about it.

In addition, we modify the HistoryEntry weight / priority so that
additions to it come after the additions to the resource off of which it
is based. As a result, we need to DEFER some foreign keys so that we can
write the billing / poll message objects before the history object that
they're referencing.
2021-04-12 12:11:20 -04:00
Lai Jiang
020273b184
Make Numulus compile on macOS (#1070)
* Make Numulus compile on macOS

BSD sed behaves differently than Linux sed. By adding a "-e" flag the
comand works in both systems.

See: https://unix.stackexchange.com/questions/101059/sed-behaves-different-on-freebsd-and-on-linux

* Make the regex easier to understand
2021-04-12 10:12:26 -04:00
Weimin Yu
0156a29f93
Try again to fix a flaky test (#1066)
* Try again to fix a flaky test

Fix DeleteExpiredDomainsActionTest.test_deletesThreeDomainsInOneRun
2021-04-08 11:47:35 -04:00
gbrodman
0b520f3885
Partially convert EppResourceUtils to SQL (#1060)
* Partially convert EppResourceUtils to SQL

Some of the rest will depend on b/184578521.

The primary conversion in this PR is the change in
NameserverLookupByIpCommand as that is the only place where the removed
EppResourceUtils method was called. We also convert to DualDatabaseTest
the tests of the callers of NLBIC. and use a CriteriaQueryBuilder in the
foreign key index SQL lookup (allowing us to avoid the String.format
call).
2021-04-07 19:20:13 -04:00
Weimin Yu
da6d90755e
Add a wipeout action for Datastore in QA (#1064)
* Add a wipeout action for Datastore in QA
2021-04-07 16:17:51 -04:00
Weimin Yu
4d04e4fd15
Add -r when rsync a release to the live folder (#1063)
* Add -r when rsync a release to the live folder

Release folders now are no longer flat. Each of them has a 'beam'
subfolder with pipeline metadata files.
2021-04-07 10:07:00 -04:00
Weimin Yu
928b272d89
Remove SQL credentials from Keyring (#1059)
* Remove SQL credentials from Keyring

Remove SQL credentials from Keyring. SQL credentials will be managed by
an automated system (go/dr-sql-security) and the keyring is no longer a
suitable place to hold them.

Also stopped loading SQL credentials from they keyring for comparison
with those from the secret manager.
2021-04-07 10:05:59 -04:00
Ben McIlwain
e31f0cb9ba
Don't send email notification when 0 uploads were attempted (#1058)
* Don't send email notification when 0 uploads were attempted
2021-04-06 18:17:57 -04:00
Michael Muller
06b0887c51
Convert RefreshDnsOnHostRenameAction to tm (#1053)
* Convert RefreshDnsOnHostRenameAction to tm

This is not quite complete because it also requires the conversion of a
map-reduce which is in scope for an entirely different work.  Tests of the
map-reduce functionality are excluded from the SQL run.

This also requires the following additional fixes:

-  Convert Lock to tm, as doing so was necessary to get this action to work.
   As Lock is being targeted as DatastoreOnly, we convert all calls in it to
   use ofyTm()
-  Fix a bug in DualDatabaseTest (the check for an AppEngineExtension field is
   wrong, and captures fields of type Object as AppEngineExtension's)
-  Introduce another VKey.from() method that creates a VKey from a stringified
   Ofy Key.

* Rename VKey.from(String) to fromWebsafeKey

* Throw NoSuchElementE. instead of NPE
2021-04-06 14:28:30 -04:00
Lai Jiang
73dcb4de4e
Enable cross referencing for generated sources (#1057)
This change should allow generated classes like AutoValue or Dagger
classes to be cross-referencable on cs.nomulus.foo

See b/184284124 for context.
2021-04-06 10:35:20 -04:00
Weimin Yu
9dd08c48bc
Use credential in secretmanager to deploy schema (#1055)
* Use credential in secretmanager to deploy schema

Fetch the schema_deployer credential from SecretManager when deploying
the schema to Cloud SQL.
2021-04-06 09:43:15 -04:00
sarahcaseybot
eabf056f9b
Correctly get the primary database value in PremiumListDualDao (#1052)
* Correctly get the primary database value in PremiumListDualDao

* Remove extra AppEngineExtension

* get rid of ofy call

* Remove extra duration skip in test
2021-04-05 13:44:30 -04:00
gbrodman
7c3ef52026
Convert poll-message-related classes to use SQL as well (#1050)
* Convert poll-message-related classes to use SQL as well

Two relatively complex parts. The first is that we needed a small
refactor on the AckPollMessagesCommand because we could theoretically be
acking more poll messages than the Datastore transaction size boundary.
This means that the normal flow of "gather the poll messages from the DB
into one collection, then act on it" needs to be changed to a more
functional flow.

The second is that acking the poll message (deleting it in most cases)
reduces the number of remaining poll messages in SQL but not in
Datastore, since in Datastore the deletion does not take effect until
after the transaction is over.
2021-04-02 19:57:26 -04:00
sarahcaseybot
75e74f013d
Add a getReservedList command (#1041)
* Add a getReservedList command

* add tests

* Remove multiple lists parameter

* print error to stderr
2021-04-02 19:23:36 +00:00
gbrodman
c077aca433
Convert AuthenticatedRegAccessor and OteStats to SQL (#1039)
This required adding a new HistoryEntryDao method but it's fairly
similar to the ones we already have.
2021-04-02 11:41:26 -04:00
gbrodman
4e7dd7a95a
Convert DomainTCF and DomainContent to tm() (#1046)
Note: this also includes conversions of the tests of any class that
called the converted DomainContent method to make sure that we caught
everything.
2021-04-02 11:41:00 -04:00
sarahcaseybot
8952687207
Add CommandWithRemoteApi to DeleteReservedListCommand (#1051) 2021-04-01 21:32:40 -04:00
Ben McIlwain
0164bceb95
Fix some low-hanging code quality issue fruits (#1047)
* Fix some low-hanging code quality issue fruits

These include problems such as: use of raw types, unnecessary throw clauses,
unused variables, and more.
2021-04-01 18:04:21 -04:00
Michael Muller
dc51019fd2
Convert ofy -> tm for two more classes (#1049)
* Convert ofy -> tm for two more classes

Convert ofy -> tm for MutatingCommand and DedupeOneTimeBillingEventIdsCommand.

Note that DedupeOneTimeBillingEventIdsCommand will not be needed after
migration, so this conversion is just to remove the ofy uses from the
codebase.  We don't update the test (other than to keep it working) and it
wouldn't currently work in SQL.

* Fixed a test broken by this PR
2021-04-01 07:27:43 -04:00
gbrodman
36762b5e08
Convert ResaveEntityAction and RelockDomainAction to tm() (#1048)
In addition, we move the deleteTestDomain method to DatabaseHelper since
it'll be useful in other places (e.g. RelockDomainActionTest) and remove
the duplicate definition of ResaveEntityAction.PATH.

We also can ignore deletions of non-persisted entities in the JPA
transaction manager.
2021-03-31 15:52:25 -04:00
gbrodman
c9980fcdec
Update RegistrarSettingsAction and RegistrarContact to SQL calls (#1042)
* Update RegistrarSettingsAction and RegistrarContact to SQL calls

Relevant potentially-unclear changes:
- Making sure the last update time is always correct and up to date in
the auto timestamp object
- Reloading the domain upon return when updating in a new transaction to
make sure that we use the properly-updated last update time (SQL returns
the correct result if retrieved within the same txn but DS does not)
2021-03-30 16:41:26 -04:00
gbrodman
d30ab08f6d
Convert DomainTAF and DomainFlowUtils to SQL (#1045)
* Convert DomainTAF and DomainFlowUtils to SQL

The only tricky part to this is that the order of entities that we're
saving during the DomainTransferApproveFlow matters -- some entities
have dependencies on others so we need to save the latter first. We
change `entitiesToSave` to be a list to reinforce this.
2021-03-30 16:33:35 -04:00