* Implement a util class to manage push queues using Cloud Tasks API
Push queues were part of App Engine when they debuted. As a result the
Task Queue API were part of the App Engine SDK and can only be used in
App Engine classic runtime. The new Cloud Tasks API can be used in any
runtime but it only supports push queues. In this PR we implement a util
class (CloudTasksUtils) like TaskQueueUtils to handle enqueuing tasks to
push queues using Cloud Tasks. One action (TldFanoutAction) was
converted to use the new API as a demo. Mass migration of other call sites of
the old API will follow in a separate PR.
TESTED=deployed to alpha and verified that tasks are corrected enqueued
and executed.
The API provided by the GAE SDK will not be available outside GAE
runtime. This presents a problem when we migrate off of GAE. More
pressingly, the RDE pipeline migration to Beam requires that we write to
GCS on GCE. Previously we were able to sidestep the issue by delegating
the writes to FileIO provided by Beam, which knows how to write to GCS.
However the RDE pipeline cannot use FileIO directly as it needs to write
to multiple files in one go and explicit use of GCS API is needed.
An unfortunate side effect of the API migration is that the new testing
library contains a bug which makes serializing GcsUtils impossible. It
is fixed upstream but not released yet. The fix has been backported for
the time being.
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1228)
<!-- Reviewable:end -->
* Update GCL dependency to avoid security alert
This required a few changes in addition to the dependency update.
- a few transitive / required dependency updates as well
- updating soyutils_usegoog.js and adding checks.js because they're
necessary as part of the Soy compilation process
- Using a trustedResourceUri in the buildSrc Soy compilation instead of
a string
- changing the arguments to the Soy-to-Java compiler to comply with the
new version
- Moving all Soy UI files to be in the registrar directory. This was
not the case before due to previous thinking that we'd have separate
admin and registrar consoles -- this is no longer the case so it's no
longer necessary. This necessitated various refactorings and reference
changes.
- The new soy-to-javascript compiler requires this, as it removes the
"deps" param that we were previously using to say "use the general UI
utils as dependencies for the registrar-console files".
- Creating a SQL environment and loading test data in the test server
main method -- previously, the local test server did not work.
- Fix some JS code that was referencing now-deleted library functions
- Removal of the Karma tests, as the karma-closure library hasn't been
updated since 2018 and it no longer works. We never noticed any errors
from the Karma tests, we never change the JS, and we have the
Java+Selenium screenshot differ tests to test the UI anyway.
* Upgrade testcontainers to work around a race
testcontainers 1.15.? has a race condition that occassionally causes deadlocks.
This can be worked around by upgrading to 1.15.2 and set transport type to
http5.
See https://github.com/testcontainers/testcontainers-java/issues/3531
for more information.
There are two changes that are not lockfiles:
- dependencies.gradle
- java_common.gradle
* Update more dependencies to newer versions
* Add lockfiles and back out 2 problematic dep updates
* Fix the build (backs out more changes)
* Back out qdox 2.0 too
* Update a few plugins for Java 11 compatibility
Guice 5.0.1 is now compatible with Java 11. However we don't
directly depend on Guice. Rather Soy depends on Guice. So I added a
direct dependency on Guice 5.0 just before Soy in order to frontload Soy
and pull in the newer version.
Mockito 3.7.7 is now compatible with Java 11. The complication is that
we need to use the inline version of Mockito, which among other things
also allows mocking for final classes (hooray!). It will eventually
become the default Mockito mock maker but for now it needs to be
manually activated.
Note that the inline version now introduces another warning:
```
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
```
Which I think is WAI due to how the inline mock maker works. Waiting on
the author to confirm.
After these to changes the only illegal reflective access is caused by
App Engine SDK tools, which we will rid ourselves of when we migrate off
of GAE.
* Restore package-lock.json
* Properly set up JPA in BEAM workers
Sets up a singleton JpaTransactionManger on each worker JVM for all
pipeline nodes to share.
Also added/updated relevant dependencies. The BEAM SDK version change
caused the InitSqlPipeline's graph to change.
* Upgrade error-prone to 3.3.4
This would fix the failure with openjdk 11.0.9 in
3.3.3.
Fixed new antipatterns raised by the new version:
- Replaced unnecessary lambdas with methods.
- Switched wait/sleep calls to equivalent methods using java.time types
- Types inheriting Object.toString() should not be assigned to string
parameter in logging statements.
* CertificateChecker with checks for expiration and key length
* Add validity length check
* Get rid of hard-coded constants and DSA checks
* add files that for some reason weren't included in last commit
* Rename violations and other fixes
* Add displayMessage to CertificateViolation enum
* Switch violations from an enum to a class
* small changes
* Get rid of ECDSA checks
* add checks for old validity length
* Change error message for validity length
Without it we kept getting the following warning:
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
For some inexplicable reasons I have to move the javax.mail package one
spot up to avoid its classes being shadowed by those provided in the
appengine package...
* Update BEAM SDK to work with Java 11
Upgraded BEAM dependencies to 2.23.0.
Updated Spec11 and invoice pipelines:
- Added the required region parameter.
- Removed the workaround code for staging.
Verified that staging is successful in alpha:
./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_spec11_pipeline --project domain-registry-alpha'
and
./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_invoicing_pipeline'
* Enable Java 11 features
As of this commit Java 11 must be used to build. The generated bytecode
is still at Java 8 due to App Engine task queue limit.
Also fixed a bug where the included google-java-format jar file is not
used, requiring the user to install it separately.
See: https://cloud.google.com/appengine/docs/standard/java/taskqueue/push
* Add use of interval data type
* Add support for Millis
* Use Java-object type
* Change column type for relock_duration
* add years and months
* Add tests for hours, minutes, and seconds
* Add javadoc describing how joda duration is stored
* Add test for lots of days
1. The Gradle apt plugin is no longer needed to process annotations.
2. Without the apt plugin, Gralde puts the source files generated by
annotation processors in build/generated/sources/annotationProcessor.
3. Change the location of custom generated files to be consistent.
4. Fix a javadoc formatting error.
* Upgrade App Engine and webserver tests from JUnit 4 to 5
* Fix most errors
* Merge branch 'master' into junit5ification
* Fix test server by extracting non-test setup/tear-down
* Merge branch 'master' into junit5ification
* Fix backup tests
* Don't createFile(); asCharSink does it
* Increase the timeout for all WebDriver tests to 60s (helps w/ flakiness)
* Add JUnit Params and start using it
* Convert rest of RDE tests
* Don't check headers for generated tests
* Expand visibility to fix build breakage
* Bump JUnit versions to 5.6.2
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.
ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.
Added a mockito dependency, and upgraded mockito-core to 3.3.3.
Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
* Add testcontainers' Junit5 support dependency
Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.
Migrated PersistenceModuleTest as an example.
Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
* Add Test suite support for JUnit 5 classes
Added Gradle dependencies and updated lockfiles.
Updated SqlInegrationTestSuite to use new annotations.
Migrated one member class in SqlIntegrationTestSuite (CursorDaoTest)
to JUnit 5, and verified that the new Suite runner can handle a
mixture of JUnit 4 and 5 tests in one suite.
Note that Gradle tests that run TestSuites must choose JUnit 4.
Updated core/build.gradle and integration/build.gradle.
* Allow backwards compatibility with JUnit 4 @Rules in JUnit 5
This allows us to defer having to re-implement all of our JUnit 4 Rules as JUnit
5 extensions for now, while continuing to in-place upgrade all existing JUnit 4
test classes to JUnit 5.
As proof of concept, this upgrades PremiumListUtils (which uses AppEngineRule,
our largest and most complicated @Rule) to use the JUnit 5 test runner.
* Apply formatter to entire file
* Start using JUnit 5
This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration
In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.
* Delete no longer needed lockfiles
* Merge branch 'master' into first-junit5
* Implement dump_golden_schema command in devtool
Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.
Refactor the SQL container functionality from GenerateSqlCommand. There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.
* Reformatted and changes in response to review
* Fix getDockerTag() usage
* Fix "leaked resource"
* Upgradle JUnit to 4.13
Removed third_party/junit folder and all usage of the
JunitBackPort class. As a result, third_party is no
longer a Gradle subproject.
Minor code changes were needed to work around an
error-prone pattern: multiple statement in assertThrows'
runnable lambda.
Also third_party/activation and third_party/jsch. These
dependencies are loaded from remote maven repo. The local
copies are not in use.
* Consolidate certificate supplier module
Both the proxy and the proxy needs certificate suppliers. The PR
consolidates the module that providings those bindings to a shared
module and switched the proxy to use that module. The prober currently
uses P12 file to store its certificates. I am debating keeping that
supplier ro converting them to PEM files for simplicity.
* Rename mode enum values to be more descriptive
* Update annotation names to be more descriptive
* Create a new app to hold GenerateSqlSchemaCommand
GenerateSqlSchemaCommand starts postgresql using testcontainer.
This makes junit etc a runtime dependency, allowing them to get
into release artifacts.
By moving this command to a separate tool, we can remove junit
etc as compile/runtime dependency.
* Fix sql script name conflict
There are two V11__ files due to concurrent merge. Renamed one
to V12__
Also removed a @NotNull annotation, which is the fist in the code base.
Most of the code base use @Nullable instead. If we do want to use
@NotNull, we may want to use the javax one instead.
* Verify RegistryTool can instantiate
Add a task that instantiates all command classes in RegistryTool
with runtimeClasspath.
Also make sure that runtimeClasspath is a superset of
compileClasspath.
* Adding junit back into the runtime classpath
Unfortunately, GenerateSqlSchemaCommand depends on junit via testcontainers.
We should really move GenerateSqlSchemaCommand out of nomulus tool (we only
use it during development) but this gets nomulus tool working for the time
being.
* Removed unnnecessary trace line.
* lockfiles generated after update_dependency.sh
* Allow project dependency to use runtimeClasspath
Project dependency should use runtimeClasspath. However, if
left unspecified, it uses 'default', which is the same as
the legacy 'runtime' configuration. (runtimeOnly dependencies
are left out).
Since runtimeClasspath cannot be referenced directly, we use
a custom config (deploy_jar) as a proxy.
By excluding testjars (leaked into 'compile' by third-party
dependencies) from runtimeClasspath, we prevent them from
getting into release artifacts.
Two meaningful changes in appengine_war.gradle and java_common.gradle
TESTED=Diffed contents of services/{module}/build/exploded-*
Only three jars are removed: hamcrest-core, junit, and
mockito-core.
