Commit graph

3668 commits

Author SHA1 Message Date
Weimin Yu
98d259449b
Use lazy injection in SendEscrow command (#1086)
* Use lazy injection in SendEscrow command

The injected object in SendEscrowReportToIcannCommand creates Ofy keys
in its static initialization routine. This happens before the RemoteApi
setup. Use lazy injection to prevent failure.
2021-04-15 16:15:01 -04:00
gbrodman
1cc8af4acd
Specify explicit ofyTm usage in SetDatabaseTransitionScheduleCommand (#1081)
* Specify explicit ofyTm usage in SetDatabaseTransitionScheduleCommand

We cannot use the standard MutatingCommand because the DB schedule is
explicitly always stored in Datastore, and once we transition to
SQL-as-primary, MutatingCommand will stage the entity changes to SQL.

In addition, we remove the raw ofy() call from the test.
2021-04-15 11:59:04 -04:00
Rachel Guan
fbef643488
make transitionId a required parameter (#1083) 2021-04-15 10:42:15 -04:00
Lai Jiang
2161e46a4b
Fix a typo (#1085) 2021-04-15 08:15:31 -04:00
Lai Jiang
d7f27bdad3
Update the gradle appengine plugin (#1082) 2021-04-14 19:33:55 -04:00
sarahcaseybot
78e139b2c8
Add a ComparePremiumLists command (#1056)
* Add a ComparePremiumLists command

* Add a command description

* fix output

* Fix comment format

* Add periods

* Small output message change

* Inline getting stdout

* Use sets

* Inline Sets.difference
2021-04-14 18:10:47 -04:00
gbrodman
87d511d5e3
Convert more classes to using SQL / TM (#1067)
* Convert more classes to using SQL / TM

Nothing much particularly crazy here
2021-04-14 16:45:06 -04:00
sarahcaseybot
eff79e9c99
Remove unecessary ClaimsList in FlowTest (#1077) 2021-04-14 13:49:35 -04:00
Weimin Yu
bb453b1982
Migrate Keyring secrets to Secret Manager (#1072)
* Migrate Keyring secrets to Secret Manager

Implented dual-read of Keyring secrets with Datastore as primary.

Implemented dual-write of keyring secrets with Datastore as primary.
Secret manager write failures are simply thrown. This is fine since all
keyring writes are manual, throught eh update_kms_keyring command.

Added a one-way migration command that copies all data to secret manager
(unencrypted).
2021-04-14 10:17:33 -04:00
Weimin Yu
8b41b5c76f
Upgrade testcontainers to work around a race (#1080)
* Upgrade testcontainers to work around a race

testcontainers 1.15.? has a race condition that occassionally causes deadlocks.
This can be worked around by upgrading to 1.15.2 and set transport type to
http5.

See https://github.com/testcontainers/testcontainers-java/issues/3531
for more information.

There are two changes that are not lockfiles:
- dependencies.gradle
- java_common.gradle
2021-04-14 09:45:09 -04:00
Lai Jiang
881f0f5f09
Make cross referencing work in Kythe, take 2 (#1079)
* Make cross referencing work in Kythe, take 2

Per suggestions on b/184284124.
2021-04-14 09:13:05 -04:00
Weimin Yu
abe6a193a8
Add hoc tool to fix duplicate contactId (#1076)
* Add hoc tool to fix duplicate contactId
2021-04-13 22:29:22 -04:00
gbrodman
d35460f14c
Convert TmchCrl and ServerSecret to cleaner tm() impls (#1068)
* Convert TmchCrl and ServerSecret to cleaner tm() impls

When I implemented this originally I knew a lot less than I know now
about how we'll be storing and retrieving these singletons from SQL. The
optimal way here is to use the single SINGLETON_ID as the primary key,
that way we always know how to create the key that we can use in the
tm() retrieval.

This allows us to use generic tm() methods and to remove the handcrafted
SQL queries.
2021-04-13 20:50:07 -04:00
gbrodman
245e2ea5a8
Enforce consistency in non-cached FKI loads (#1075)
* Enforce consistency in non-cached FKI loads

For the cached code path, we do not require consistency but we do
require the ability to load / operate on large numbers of entities (so,
we must do so without a Datastore transaction). For the non-cached code
path, we require consistency but do not care about large numbers of
entities, so we must remain in the transaction that we're already in.
2021-04-13 15:14:02 -04:00
sarahcaseybot
65f35ac8c1
Fix TimestampInversionException (#1065)
* Fix TimestampInversionException

* Add autoIncrement

* unset auto increment mode
2021-04-13 11:59:14 -04:00
sarahcaseybot
994af085d8
Add a CompareReservedListCommand (#1054)
* Add a CompareReservedListCommand

* compare maps

* output format fixes

* Clean up loops

* Inline Sets.difference()

* Remove ImmutableCopy()
2021-04-13 11:45:45 -04:00
Lai Jiang
ce25cea134
Disable TLS tests related to v1.1 (#1074)
There is no need for this test now because we've past the enforcement
date. We should take out the entire enforcement date logic but right now
this test is failing because TLS 1.1 is not being supported anymore by
the latest release of JDK 11.

The other test is a bit tricky to fix, see comment.

Disable these tests for now to unblock development.
2021-04-13 10:30:58 -04:00
gbrodman
92dcacf78c
Add a beforeSqlSave callback to ReplaySpecializer (#1062)
* Add a beforeSqlSave callback to ReplaySpecializer

When in the Datastore-primary and SQL-secondary stage, we will want to
save the EppResource-at-this-point-in-time field in the *History
objects so that later on we can examine the *History objects to see what
the resource looked like at that point in time.

Without this PR, the full object at that point in time would be lost
during the asynchronous replay since Datastore doesn't know about it.

In addition, we modify the HistoryEntry weight / priority so that
additions to it come after the additions to the resource off of which it
is based. As a result, we need to DEFER some foreign keys so that we can
write the billing / poll message objects before the history object that
they're referencing.
2021-04-12 12:11:20 -04:00
Lai Jiang
020273b184
Make Numulus compile on macOS (#1070)
* Make Numulus compile on macOS

BSD sed behaves differently than Linux sed. By adding a "-e" flag the
comand works in both systems.

See: https://unix.stackexchange.com/questions/101059/sed-behaves-different-on-freebsd-and-on-linux

* Make the regex easier to understand
2021-04-12 10:12:26 -04:00
Weimin Yu
0156a29f93
Try again to fix a flaky test (#1066)
* Try again to fix a flaky test

Fix DeleteExpiredDomainsActionTest.test_deletesThreeDomainsInOneRun
2021-04-08 11:47:35 -04:00
gbrodman
0b520f3885
Partially convert EppResourceUtils to SQL (#1060)
* Partially convert EppResourceUtils to SQL

Some of the rest will depend on b/184578521.

The primary conversion in this PR is the change in
NameserverLookupByIpCommand as that is the only place where the removed
EppResourceUtils method was called. We also convert to DualDatabaseTest
the tests of the callers of NLBIC. and use a CriteriaQueryBuilder in the
foreign key index SQL lookup (allowing us to avoid the String.format
call).
2021-04-07 19:20:13 -04:00
Weimin Yu
da6d90755e
Add a wipeout action for Datastore in QA (#1064)
* Add a wipeout action for Datastore in QA
2021-04-07 16:17:51 -04:00
Weimin Yu
4d04e4fd15
Add -r when rsync a release to the live folder (#1063)
* Add -r when rsync a release to the live folder

Release folders now are no longer flat. Each of them has a 'beam'
subfolder with pipeline metadata files.
2021-04-07 10:07:00 -04:00
Weimin Yu
928b272d89
Remove SQL credentials from Keyring (#1059)
* Remove SQL credentials from Keyring

Remove SQL credentials from Keyring. SQL credentials will be managed by
an automated system (go/dr-sql-security) and the keyring is no longer a
suitable place to hold them.

Also stopped loading SQL credentials from they keyring for comparison
with those from the secret manager.
2021-04-07 10:05:59 -04:00
Ben McIlwain
e31f0cb9ba
Don't send email notification when 0 uploads were attempted (#1058)
* Don't send email notification when 0 uploads were attempted
2021-04-06 18:17:57 -04:00
Michael Muller
06b0887c51
Convert RefreshDnsOnHostRenameAction to tm (#1053)
* Convert RefreshDnsOnHostRenameAction to tm

This is not quite complete because it also requires the conversion of a
map-reduce which is in scope for an entirely different work.  Tests of the
map-reduce functionality are excluded from the SQL run.

This also requires the following additional fixes:

-  Convert Lock to tm, as doing so was necessary to get this action to work.
   As Lock is being targeted as DatastoreOnly, we convert all calls in it to
   use ofyTm()
-  Fix a bug in DualDatabaseTest (the check for an AppEngineExtension field is
   wrong, and captures fields of type Object as AppEngineExtension's)
-  Introduce another VKey.from() method that creates a VKey from a stringified
   Ofy Key.

* Rename VKey.from(String) to fromWebsafeKey

* Throw NoSuchElementE. instead of NPE
2021-04-06 14:28:30 -04:00
Lai Jiang
73dcb4de4e
Enable cross referencing for generated sources (#1057)
This change should allow generated classes like AutoValue or Dagger
classes to be cross-referencable on cs.nomulus.foo

See b/184284124 for context.
2021-04-06 10:35:20 -04:00
Weimin Yu
9dd08c48bc
Use credential in secretmanager to deploy schema (#1055)
* Use credential in secretmanager to deploy schema

Fetch the schema_deployer credential from SecretManager when deploying
the schema to Cloud SQL.
2021-04-06 09:43:15 -04:00
sarahcaseybot
eabf056f9b
Correctly get the primary database value in PremiumListDualDao (#1052)
* Correctly get the primary database value in PremiumListDualDao

* Remove extra AppEngineExtension

* get rid of ofy call

* Remove extra duration skip in test
2021-04-05 13:44:30 -04:00
gbrodman
7c3ef52026
Convert poll-message-related classes to use SQL as well (#1050)
* Convert poll-message-related classes to use SQL as well

Two relatively complex parts. The first is that we needed a small
refactor on the AckPollMessagesCommand because we could theoretically be
acking more poll messages than the Datastore transaction size boundary.
This means that the normal flow of "gather the poll messages from the DB
into one collection, then act on it" needs to be changed to a more
functional flow.

The second is that acking the poll message (deleting it in most cases)
reduces the number of remaining poll messages in SQL but not in
Datastore, since in Datastore the deletion does not take effect until
after the transaction is over.
2021-04-02 19:57:26 -04:00
sarahcaseybot
75e74f013d
Add a getReservedList command (#1041)
* Add a getReservedList command

* add tests

* Remove multiple lists parameter

* print error to stderr
2021-04-02 19:23:36 +00:00
gbrodman
c077aca433
Convert AuthenticatedRegAccessor and OteStats to SQL (#1039)
This required adding a new HistoryEntryDao method but it's fairly
similar to the ones we already have.
2021-04-02 11:41:26 -04:00
gbrodman
4e7dd7a95a
Convert DomainTCF and DomainContent to tm() (#1046)
Note: this also includes conversions of the tests of any class that
called the converted DomainContent method to make sure that we caught
everything.
2021-04-02 11:41:00 -04:00
sarahcaseybot
8952687207
Add CommandWithRemoteApi to DeleteReservedListCommand (#1051) 2021-04-01 21:32:40 -04:00
Ben McIlwain
0164bceb95
Fix some low-hanging code quality issue fruits (#1047)
* Fix some low-hanging code quality issue fruits

These include problems such as: use of raw types, unnecessary throw clauses,
unused variables, and more.
2021-04-01 18:04:21 -04:00
Michael Muller
dc51019fd2
Convert ofy -> tm for two more classes (#1049)
* Convert ofy -> tm for two more classes

Convert ofy -> tm for MutatingCommand and DedupeOneTimeBillingEventIdsCommand.

Note that DedupeOneTimeBillingEventIdsCommand will not be needed after
migration, so this conversion is just to remove the ofy uses from the
codebase.  We don't update the test (other than to keep it working) and it
wouldn't currently work in SQL.

* Fixed a test broken by this PR
2021-04-01 07:27:43 -04:00
gbrodman
36762b5e08
Convert ResaveEntityAction and RelockDomainAction to tm() (#1048)
In addition, we move the deleteTestDomain method to DatabaseHelper since
it'll be useful in other places (e.g. RelockDomainActionTest) and remove
the duplicate definition of ResaveEntityAction.PATH.

We also can ignore deletions of non-persisted entities in the JPA
transaction manager.
2021-03-31 15:52:25 -04:00
gbrodman
c9980fcdec
Update RegistrarSettingsAction and RegistrarContact to SQL calls (#1042)
* Update RegistrarSettingsAction and RegistrarContact to SQL calls

Relevant potentially-unclear changes:
- Making sure the last update time is always correct and up to date in
the auto timestamp object
- Reloading the domain upon return when updating in a new transaction to
make sure that we use the properly-updated last update time (SQL returns
the correct result if retrieved within the same txn but DS does not)
2021-03-30 16:41:26 -04:00
gbrodman
d30ab08f6d
Convert DomainTAF and DomainFlowUtils to SQL (#1045)
* Convert DomainTAF and DomainFlowUtils to SQL

The only tricky part to this is that the order of entities that we're
saving during the DomainTransferApproveFlow matters -- some entities
have dependencies on others so we need to save the latter first. We
change `entitiesToSave` to be a list to reinforce this.
2021-03-30 16:33:35 -04:00
gbrodman
b90b9af80e
Convert RDE classes to use tm() (#1044)
This is mostly just using the generic Cursor load methods with the
slight difference that before we relied on ofy() returning null on
absent entities.
2021-03-30 13:09:33 -04:00
gbrodman
5c6b2595db
Convert Kms* classes to use SQL when appropriate (#1043)
* Convert Kms* classes to use SQL when appropriate
2021-03-29 18:03:42 -04:00
gbrodman
e30d3efa7c
Convert DomainCreateFlow to use generic tm() methods (#1026)
Various necessary changes included as part of this:

- Make ForeignKeyIndex completely generic. Previously, only the load()
method that took a DateTime as input could use SQL, and the cached flow
was particular to Objectify Keys. Now, the cached flow and the
non-cached flow can use the same (ish) piece of code to load / create
the relevant index objects before filtering or modifying them as
necessary.
- EntityChanges should use VKeys
- FlowUtils should persist entity changes using tm(), however not all
object types are storable in SQL.
- Filling out PollMessage fields with the proper object type when
loading from SQL
- Changing a few tm() calls to ofyTm() calls when using objectify. This
is because creating a read-only transaction in SQL is quite a footgun at
the moment, because it makes the entire transaction you're in (if you
were already in one) a read-only transaction.
2021-03-29 15:39:32 -04:00
Michael Muller
db26635825
Convert 3 classes from ofy -> tm (#1034)
* Convert 3 classes from ofy -> tm

Convert SyncGroupMembersAction, SyncRegistrarsSheet and
IcannReportingUploadAction and their test cases to use TransactionManager and
dual-test them so we know they work in jpa.

* Address comments in review

Address review comments and make the entire IcannReportingUploadAction run
transactional.

* reformatted.

* Remove duplicate loadByKey() method

Remove test method added in a recent PR.
2021-03-29 13:08:15 -04:00
gbrodman
65e468f2bc
Update ListDomainsAction to SQL (#1036) 2021-03-29 12:54:45 -04:00
gbrodman
a4e078305d
Embed a ZonedDateTime as the UpdateAutoTimestamp in SQL (#1033)
* Embed a ZonedDateTime as the UpdateAutoTimestamp in SQL

This means we can get rid of the converter and more importantly, means
that reading the object from SQL does not affect the last-read time (the
test added to UpdateAutoTimestampTest failed prior to the production
code change).

For now we keep both time fields in UpdateAutoTimestamp however
post-migration, we can remove the joda-time field if we wish.

Note: I'm not sure why <now> is the time that we started getting
LazyInitializationExceptions in the LegacyHistoryObject and
ReplayExtension tests but we can solve that by just examining /
initializing the object within the transaction.
2021-03-29 11:59:08 -04:00
gbrodman
1e650bd0a1
Convert to tm() some low-hanging ofy fruit (#1029)
* Convert to tm() some low-hanging ofy fruit
2021-03-25 20:01:53 -04:00
Ben McIlwain
2649a9362a
Make TransactionManager.loadAllOf() smart w.r.t the cross-TLD entity group (#1040)
* Make TransactionManager.loadAllOf() smart w.r.t the cross-TLD entity group

The loadAllOf() method will now automatically append the cross-TLD entity group
ancestor query as necessary, iff the entity class being loaded is tagged with
the new @IsCrossTld annotation.

* Add tests
2021-03-25 18:55:18 -04:00
Weimin Yu
3c65ad0f8a
Add SQL wipeout action in QA (#1035)
* Add SQL wipeout action in QA

Added the WipeOutSqlAction that deletes all data in Cloud SQL.

Wipe out is restricted to the QA environment, which will get production
data during migration testing.

Also added a cron job that invokes wipeout on every saturday morning.
This is part of the privacy requirments for using production data in QA.

Tested in QA.
2021-03-25 16:59:09 -04:00
Ben McIlwain
2bfd02f977
Add some load convenience methods to DatabaseHelper (#1038)
* Add some load convenience methods to DatabaseHelper

These can only be called by test code, and they automatically wrap the load
in a transaction if one isn't already specified (for convenience).

In production code we don't want to be able to use these, as we have to be
more thoughtful about transactions in production code (e.g. make sure that
we aren't loading and then saving a resource in separate transactions in a
way that makes it prone to contention errors).
2021-03-25 16:14:46 -04:00
Ben McIlwain
3af0f8c148
Include ReservedList in BigQuery exports (#1037)
* Include ReservedList in BigQuery exports
2021-03-25 13:14:29 -04:00