* Rename client ID to registrar ID in most places
This is a code-only change, that shouldn't require any sort of data
migration. Correspondingly, there are some existing uses of clientId that are
not migrated (e.g. Datastore fields, task queue payloads, URL parameters for
actions that might be hit from task queues, etc.). And it of course doesn't
modify any fields in EPP XML. Note that the Cloud SQL schema fields are
already named using the registar_id pattern.
This also doesn't yet touch on the -c parameters in nomulus tools; that will be
coming later (since that is an external manual touch-point, it will require a
lot more in the way of changes to various meta scripts and documentation).
* Change more client IDs
* Merge branch 'master' into clientid-to-registrarid
* Rename Spec11Pipeline's Subdomain -> DomainNameInfo
"Subdomain" never made any sense as a class name because these are all
second-level domain names, along with a little bit of metadata such as some
registrar info. "DomainNameInfo" is a better fit.
* Update references to RDAP RFCs
There were minor changes to the RDAP RFCs used -- we don't need to
change anything since we already comply with all of the changes, but we
should refer to the newer RFCs in the code.
* Add a temporary fix to Hibernate detach in query
Make all queries in RegistryQuery (exclusively used by BEAM) use
EntityManager.clear() to detach entities. This is a temporary measure
that unblocks work in BEAM. We will revert the work once
JpaTransactionManager can detach entities properly for all types of
queries.
Also fixed regression bugs that broke query result streaming:
- The code that sets query fetch size was not carried over from
QueryComposer.
- The new ReadOnlyCheckingTypedQuery class did not override parent's
getResultStream() method, which calls getList().
* Add the DNS refresh request time field to the Domain tables
This isn't used yet, but it will eventually be the replacement for the dns-pull
task queue once we get further in the migration.
* Merge branch 'master' into domain-dns-dirty
* Remove VKeyTranslatorFactory.createVKey(String)
This method serves the same function as VKey.fromWebsafeKey(), and isn't used
anywhere. Move the test for it into VKeyTest and use it to instead test
fromWebsafeKey() (which didn't previously have a test).
I'm not sure why this test is failing. It's failing saying that the
listObjects call is failing to include
"soy_2000-01-01_thin_S1_R1.xml.ghostryde" in the results, however the
verifyFiles method that we call right beforehand verifies that file and
its contents
This includes a change to how the JPA transaction manager handles
existence and load checks for entities with compound IDs. Previously, we
relied on the fields all being named the same in the ID entity and the
parent entity. This didn't work for History objects (e.g. DomainHistory)
so existence checks were broken. Now, we use the methods the same way
that Hibernate does (if possible).
Note as well that there's a bit of semi-duplicated logic in
DeleteProberDataAction (between the mapper and the SQL logic). The
mapper code will be deleted once we've shifted to SQL, and for now it's
better to keep it in place for logging purposes.
This involves:
- Altering both transaction managers to check for a read-only mode at
the start of standard write actions (e.g. delete, put).
- Altering both raw layers (entity manager, ofy) to throw exceptions on
write actions as well
- Implementing bypass routes for reading / setting / removing the schedule itself
so that we don't get "stuck"
* Clean up ReplicateToDatastoreAction and tests
1. applyTransaction should throw an error if it fails; this allows us to
have more information in the caller (and it shouldn't usually happen)
2. Set a response code + payload now, since this is an action that is
called by cron
3. Add a method to the test log subject that allows us to check if a
severe log with a particular Throwable cause was logged (since the cause
isn't contained in the log message itself directly)
* Save indexes when replaying EppResources SQL->DS
We implement this similarly to how we implement the
beforeSqlSaveOnReplay callback in the other direction -- a
beforeDatastoreSaveOnReplay method that is called when replaying a
Mutation to Datastore. This means that the asynchronous replay will
create the relevant ForeignKeyIndex and EppResourceIndex objects for
EppResources saved when SQL is primary.
* Implement a util class to manage push queues using Cloud Tasks API
Push queues were part of App Engine when they debuted. As a result the
Task Queue API were part of the App Engine SDK and can only be used in
App Engine classic runtime. The new Cloud Tasks API can be used in any
runtime but it only supports push queues. In this PR we implement a util
class (CloudTasksUtils) like TaskQueueUtils to handle enqueuing tasks to
push queues using Cloud Tasks. One action (TldFanoutAction) was
converted to use the new API as a demo. Mass migration of other call sites of
the old API will follow in a separate PR.
TESTED=deployed to alpha and verified that tasks are corrected enqueued
and executed.
Add double-replay to the Host*Flow tests to show how this works. The
only change to the double replay itself is that now we store the
Datastore entity in the TransactionEntity object -- this is because we
use Objectify to serialize the objects into bytes and we need it to know
about the entity in question.
* Add DS->SQL replay cron job to production
This won't do anything until we set the migration schedule to
DATASTORE_PRIMARY. Actions in order:
1. Add this cron job (it'll be a no-op)
2. Run the init-sql-pipeline to populate production's SQL DB
3. Set the SqlReplayCheckpoint to a time before the smear backup that
was used in step #1 (maybe 30 minutes)
4. Set the database migration schedule to transition to
DATASTORE_PRIMARY at some point
* Remove ReservedList from Datastore schema
* Remove some Datastore references
* Add a different non-replicated entity to ReplayCommitLogsToSqlActionTest
We shouldn't reference tm() at all before initializing the JPA
transaction manager, since tm() looks at the database migration schedule
when figuring out which transaction manager to use.
* Remove recursive load in DBMSS cache
This occurs because if we do a standard transaction, the JpaTxnManager
checks to see if we should be doing backups, which involves loading the
migration state schedule (causing the recursion). When starting the
transaction to load the schedule, we should explicitly
transactWithoutBackup so there's no need to check.
This wasn't hit in tests because we previously manually set the
replication to not occur in the JpaTransactionManagerExtension -- we
remove that and related setters.
* Generate string to uniquely identify a SqlEntity
Add a method to SqlEntity that returns a string built from the entity's
primary key(s). This string can be used in logging.
* Store DatabaseMigrationSchedule in SQL instead of Datastore
This requires messing around with some of the JPA unit test rule
creation since it requires saving / retrieving the schedule pretty much
always (which itself includes the hstore extension).
This performs a direct load-by-key (the most efficient Datastore operation),
rather than attempting to load all entities by type using an ancestor query. The
existing implementation is possibly more error-prone as well, and might be
responsible for the "cross-group transaction need to be explicitly specified"
error we're seeing.
* Remove PremiumList from Datastore schema
* Remove commented out code
* Change lastUpdateTime to creationTimestamp
* Remove extra file
* Remove currency unit from input data to parse
* Revert extra file
* Check currency in parse
* Create all PremiumEntries before saving them in bulk
* small fixes
* Fix merge conflict
There was a subtle issue that we encountered in sandbox when using one
transaction per file that was difficult to replicate. Basically,
1. Save a domain with dsData
2. Save the domain without dsData
3. Save the domain with the same dsData as step 1
4. Delete literally any object
If one performs steps 2-4 in the same transaction, Hibernate will throw
an exception (cascade re-saving a cascade-deleted object). Note that
step 4 is in fact necessary to reproduce the issue, yay Hibernate.
We will test this and if one transaction per transaction is too slow,
we'll figure out ways to reduce the number of SQL transactions.
