Various necessary changes included as part of this:
- Make ForeignKeyIndex completely generic. Previously, only the load()
method that took a DateTime as input could use SQL, and the cached flow
was particular to Objectify Keys. Now, the cached flow and the
non-cached flow can use the same (ish) piece of code to load / create
the relevant index objects before filtering or modifying them as
necessary.
- EntityChanges should use VKeys
- FlowUtils should persist entity changes using tm(), however not all
object types are storable in SQL.
- Filling out PollMessage fields with the proper object type when
loading from SQL
- Changing a few tm() calls to ofyTm() calls when using objectify. This
is because creating a read-only transaction in SQL is quite a footgun at
the moment, because it makes the entire transaction you're in (if you
were already in one) a read-only transaction.
* Convert 3 classes from ofy -> tm
Convert SyncGroupMembersAction, SyncRegistrarsSheet and
IcannReportingUploadAction and their test cases to use TransactionManager and
dual-test them so we know they work in jpa.
* Address comments in review
Address review comments and make the entire IcannReportingUploadAction run
transactional.
* reformatted.
* Remove duplicate loadByKey() method
Remove test method added in a recent PR.
* Embed a ZonedDateTime as the UpdateAutoTimestamp in SQL
This means we can get rid of the converter and more importantly, means
that reading the object from SQL does not affect the last-read time (the
test added to UpdateAutoTimestampTest failed prior to the production
code change).
For now we keep both time fields in UpdateAutoTimestamp however
post-migration, we can remove the joda-time field if we wish.
Note: I'm not sure why <now> is the time that we started getting
LazyInitializationExceptions in the LegacyHistoryObject and
ReplayExtension tests but we can solve that by just examining /
initializing the object within the transaction.
* Make TransactionManager.loadAllOf() smart w.r.t the cross-TLD entity group
The loadAllOf() method will now automatically append the cross-TLD entity group
ancestor query as necessary, iff the entity class being loaded is tagged with
the new @IsCrossTld annotation.
* Add tests
* Add SQL wipeout action in QA
Added the WipeOutSqlAction that deletes all data in Cloud SQL.
Wipe out is restricted to the QA environment, which will get production
data during migration testing.
Also added a cron job that invokes wipeout on every saturday morning.
This is part of the privacy requirments for using production data in QA.
Tested in QA.
* Add some load convenience methods to DatabaseHelper
These can only be called by test code, and they automatically wrap the load
in a transaction if one isn't already specified (for convenience).
In production code we don't want to be able to use these, as we have to be
more thoughtful about transactions in production code (e.g. make sure that
we aren't loading and then saving a resource in separate transactions in a
way that makes it prone to contention errors).
* Add Gradle tasks to stage BEAM pipelines
Add a Gracle task to stage flex-template based pipelines for alpha and
crash environments.
This is a follow up to go/r3pr/1028, which is also under review.
Add the actual release tag and beam staging project id to the config
file. This allows the Nomulus server to find the right version of the
BEAM pipelines to launch.
* Disallow admin triggering of internal endpoints
Stop simply relying on the presence of the X-AppEngine-QueueName as an
indicator that an endpoint has been triggered internally, as this allows
admins to trigger a remote execution vulnerability.
We now supplement this check by ensuring that there is no authenticated user.
Since only an admin user can set these headers, this means that the header
must have been set by an internal request.
Tested:
In addition to the new unit test, verified on Crash that:
- Internal requests are still getting authenticated via the internal auth
mechanism.
- Admin requests with the X-AppEngine-QueueName header are rejected as
"unauthorized."
* Reformatted.
* Pass --java-binary to _all_ formatter invocations
When implementing a flag to pass in the java binary to
google-java-format-diff.py, I missed the location in showNoncompliantFiles
which gets run before a check.
This change also refactors the core logic of the script so that
google-java-format-diff.py is only called from one place and (in all but one
case) only one time.
Tested:
Ran check format and show, with and without diffs present in the tree.
This has the same issue as the domain-search action where the database
ordering is not consistent between Objectify and SQL -- as a result,
there is one test that we have to duplicate in order to account for the
two sort orders.
In addition, there isn't a way to query @Convert-ed fields in Postgres
via the standard Hibernate / JPA query language, meaning we have to use
a raw Postgres query for that.
* Add a jpaTm().query(...) convenience method
This replaces the more ungainly jpaTm().getEntityManager().createQuery(...).
Note that this is in JpaTransactionManager, not the parent TransactionManager,
because this is not an operation that Datastore can support. Once we finish
migrating away from Datastore this won't matter anyway because
JpaTransactionManager will be merged into TransactionManager and then deleted.
In the process of writing this PR I discovered several other methods available
on the EntityManager that may merit their own convenience methods if we start
using them enough. The more commonly used ones will be addressed in subsequent
PRs. They are:
jpaTm().getEntityManager().getMetamodel().entity(...).getName()
jpaTm().getEntityManager().getCriteriaBuilder().createQuery(...)
jpaTm().getEntityManager().createNativeQuery(...)
jpaTm().getEntityManager().find(...)
This PR also addresses some existing callsites that were calling
getEntityManager() rather than using extant convenience methods, such as
jpa().insert(...).
* Add replay to remaining (non-trivial) flow tests
Convert all remaining flow tests to do replay/compare testing. In the course
of this:
- Move the class specific SetClock extension into its own place.
- Fix another "cyclic" foreign key (there may be another solution in this case
because HostHistory is actually different from HistoryEntry, but that would
require changing the way we establish priority since HostHistory is not
distinguished from HistoryEntry in the current methodology)
* Attempt to fix flakey deleteExpiredDomain test
Though hard to reproduce locally, the test_deletesThreeDomainsInOneRun
test has failed multiple times on Kokoro. The root cause may be the
non-transactional query executed by the Action object, which was by
design. Observing that the other test never fails, this PR follows its behavior
and adds a transactional query before invoking the action.
* Allow nom_build to run in Cloudbuild
Our builder comes with python3.6 and cannot support nom_build out of
box. Nom_build requires dataclasses which is introduced in v3.7.
I haven't found an easy way to get python3.7+ without changing the base
linux image. This PR explicitly installs dataclasses.
The dual DAO takes care of switching between databases, comparing the
results of one to the results of the other, and caching the result. All
calls to ClaimsList retrieval or storing should use the
dual-database-DAO.
Previously, calls to comparing the lists were somewhat scattered
throughout the codebase. Now, there is one class for retrieval and
comparison (the dual DAO), one class for retrieval from SQL (the SQL
DAO), and one class for retrieval from Datastore (ClaimsListShard
itself, though the retrieval could be moved in to a separate DAO if we
wished).
In addition, we rename the ClaimsListDao to ClaimsListSqlDao
* Update creation script for schema_deployer
Move the create user command for schema_deployer before the
initialization of roles. As the owner of all schema objects, it needs to
be present before grant statements are executed.
Also fixed a bug in credential printing, which fails when the password
contains '%'.
This allows us to get rid of the DAO as well as the sanity-checking
methods since we can be reasonably sure that the fields will be the
same. Future PRs will add conversions from ofy() to tm() calls that will
make sure that we get the same proper data in both Datastore and SQL
* Convert more flow tests to replay/compare
Add the replay extension to another batch of flow tests. In the course of
this:
- Refactor out domain deletion code into DatabaseHelper so that it can be used
from multiple tests.
- Make null handling uniform for contact phone numbers.
* Convert postLoad method to onLoad.
* Remove "Test" import missed during rebase
* Deal with persistence of billing cancellations
Deal with the persistence of billing cancellations, which were added in the
master branch since before this PR was initially sent for review.
* Adding forgotten flyway file
* Removed debug variable
* Add schema_deployer SQL user to SecretManager
Add the 'schema_deployer' user to the SecretManager so that its
credential can be set up. The schema deployment process will use this
user instead of the 'postgres' user.
Changed the output of the get_sql_credential command for the schema
deployment process.
Added a sql script that documents the privileges granted to
'schema_deployer'.
* Clear autorenew end time when a domain is restored
This allows us to still see in the database which now-deleted domains had
reached expiration, while correctly not re-deleting the domain immediately if
the registrar pays to explicitly restore the domain.
This also resolves some TODOs around data migration for this field on domain so
that it's not null, as said migration has already been completed.
* Allow java-format to use java from the PATH
When invoking java from the google-java-format-git-diff.sh script, if there is
no JAVA_HOME environment variable, attempt to instead run the java binary that
is on the PATH.
This also adds a few checks to verify that a java binary is available in one
of those locations and that the version discovered is Java 11 (which we know
to be compatible with the google-java-format jar).
Tested:
- unset JAVA_HOME, verified that we get the version on the PATH
- Set JAVA_HOME to an invalid directory, verified that we get an error.
- Changed the "which" command to lookup an nonexistent binary, unset JAVA_HOME
and verified that we get a "java not found" error.
- Changed the path to point to an old version of java, verified that we get a
"bad java version" error.
- Verified that the script still runs normally.
* Remove grace period ID @OnLoads now that migration is complete
I verified in BigQuery that all grace period IDs are now allocated (as expected
given that the re-save all EPP resource mapreduce has been run several times
since this migration started last year). The query I used for verification is:
SELECT fullyQualifiedDomainName, gp, ot
FROM `domain-registry.latest_datastore_export.DomainBase`
JOIN UNNEST(gracePeriods.billingEventRecurring) AS gp
JOIN UNNEST(gracePeriods.billingEventOneTime) AS ot
WHERE gp.id IS NULL or ot.id IS NULL
BUG=169873747
* Add daily cron entries to for DeleteExpiredDomainsAction
This also requires setting this action to GET instead of POST, as GAE cron makes
GET requests.
* Use shared jar to stage BEAM pipeline if possible
Allow multiple BEAM pipelines with the same classes and dependencies to
share one Uber jar.
Added metadata for BulkDeleteDatastorePipeline.
Updated shell and Cloud Build scripts to stage all pipelines in one
step.
* Add comments to Cloud SQL configs
I believe the similarity in trace to https://github.com/brettwooldridge/HikariCP/issues/1212
is misleading.
The real cause of the exceptions may be that we ran out of connections. At the
time, the production Cloud SQL server could handle 500 connections at the
maximum. That number was within reach of a busy Nomulus server.
The maximum connection in production has been increased to 1000. We
haven't encountered this issue for a long time. All connection problems
are due to Cloud SQL maintenance or other GCP related issues.
This issue is tracked by b/154720215, which is being closed with this
PR.
* Convert DomainTransferRequestFlow to tm() calls
Besides the standard ofy-to-tm conversions this includes storing the
billing event cancellation VKey in the DomainTransferData object so that
we know to handle it on process / cancellation.
* Don't use --fork-point to determine merge base
It turns out that the --fork-point option is subtle and error-prone. Its
intent is not to show the nearest common base commit, but rather the commit
on a branch that the HEAD (in this case) was originally forked off of,
_whether it is currently part of the history of the specified branch or not_
(this can happen if the branch is rewritten). The option also relies on the
presence of the fork point in the reflog for the branch, which can be
discarded in the course of a "git gc".
It is fairly easy to construct a case where the use of --fork-point causes an
error and outputs nothing. In fact, I discovered the problem as a result of
this occuring spontaneously on one of my own branches (likely related to a
rebase). Since the fork-point is empty, we end up diffing against the index
instead of the common commit.
This may have been a factor in some of the unrelated reformatting that we've
seen in past PRs.
Change this to a simple "merge-base origin/master HEAD", which outputs the
commit id of the most recent common base revision.
This change also quotes the forkPoint variable, which likely would have
resulted in an error in this case instead of silently producing the wrong
output.
* Stage the init_sql_pipeline in CloudBuild
Defined metadata file and added Gradle uberJar task for the pipeline,
which are needed for staging.
Updated cloud build script to stage this pipeline during the build
processs.
* Add TODOs regarding cloud sql database name change
We should choose a different database name for nomulus data since using
'postgres' is bad practice. See b/181693544 more background.
We have decided to delay the db change to the time when we upgrade
postgresql version. This PR adds TODOs to all occurrences of the jdbcUrl
property, including those in the internal-repo. This property will change
when we upgrade, so the TODOs will be noticed.
* Use ReplaySpecializer to fix DomainBase replays
DomainBase currently has a number of ancillary objects that require a
cascading delete that doesn't get propagated. Implement beforeSqlDelete() in
DomainContent to delete these child entities.
* Remove unnecessary Query variable
* Fix rebase error
* Update more dependencies to newer versions
* Add lockfiles and back out 2 problematic dep updates
* Fix the build (backs out more changes)
* Back out qdox 2.0 too
* Print JAVA_HOME and PATH in the java format checker task
This allows is to identify if the java version is incompatible with the
formater checker jar file.
* Show which Java binary is used
* show java version
* Fix exeInBash arguments
* Print out env variables in java format
Print out JAVA_HOME and PATH variable in the google-java-format-diff.py script
immediately prior to running the underlying java program that does the actual
format checking.
* Use the java binary from JAVA_HOME for java-format
Use "$JAVA_HOME/bin/java" for invoking the java format check instead of
whatever version of java happens to be on the path.
* Removed unused import
* Rewrite the JPA output connector for BEAM
Following BEAM's IO connector style, added a RegistryJpaIO class to hold
IO connectors, and implemented the Write connector as a static inner
class in it. The JpaTransactionManager used by the Write connector
retrieves SQL credentials from the SecretManager.
Cleaned up SQL-related pipeline parameters.
Converted the InitSqlPipeline to use RegistryJpaIO.
* Add SQL queries to RdapDomainSearchAction
Unfortunately, because ORDER BY uses the locale's sorting functionality,
we end up with some weird sort orders in SQL-land (notably, periods are
ignored / omitted). As a result, a few of the tests have to be separated
out into ofy and SQL versions based on the expected sort order.
In addition, there isn't a way to query @Convert-ed fields in Postgres
via the standard Hibernate / JPA query language, meaning we have to use
a raw Postgres query for that.
* Added "show_upgrade_diffs" script
"show_upgrade_diffs" pulls a git directory and a user branch from nomulus and
compares all of the versions of all dependencies specified in all lockfiles in
the master branch with those of the user branch and prints a nice, terse
little colorized report on the differences.
This is useful for reviewing a dependency upgrade.
* Add license header
* Changes requested in review
* Changes for review
- Change format of output so different actions are displayed somewhat
consistently.
- Make specifying a directory optional, if not specified create a temporary
directory and clean it up afterwards.
* Add a "ReplaySpecializer" to fix certain replays
Due to the fact that a given entity in either database type can map to
multiple entities in the other database, there are certain replication
scenarios that don't quite work. Current known examples include:
- propagation of cascading deletes from datastore to SQL
- creation of datastore indexed entities for SQL entities (where indexes are a
first-class concept)
This change introduces a ReplaySpecializer class, which allows us to declare
static method hooks at the entity class level that define any special
operations that need to be performed before or after replaying a mutation for
any given entity type.
Currently, "before SQL delete" is the only supported hook. A change to
DomainContent demonstrating how this facility can be used to fix problems in
cascading delete propagation will be sent as a subsequent PR.
* Throw exception on beforeSqlDelete failures
* Changes for review
