* Add Gradle tasks to stage BEAM pipelines
Add a Gracle task to stage flex-template based pipelines for alpha and
crash environments.
This is a follow up to go/r3pr/1028, which is also under review.
* Use shared jar to stage BEAM pipeline if possible
Allow multiple BEAM pipelines with the same classes and dependencies to
share one Uber jar.
Added metadata for BulkDeleteDatastorePipeline.
Updated shell and Cloud Build scripts to stage all pipelines in one
step.
* Stage the init_sql_pipeline in CloudBuild
Defined metadata file and added Gradle uberJar task for the pipeline,
which are needed for staging.
Updated cloud build script to stage this pipeline during the build
processs.
* Disable whois caching in nomulus tool
The whois commands previously served output generated from cached EppResource
objects in most cases. While this is entirely appropriate from server-side,
it is less useful when these commands are run from nomulus tool and, in fact,
when run from the "shell" command this results in changes that have been
applied from the shell not being visible from a subsequent "whois". The
command may instead serve information on an earlier, cached version of the
resource instead of the latest version.
This implementation uses dagger for parameterization of cached/non-cached
modes. I did consider the possibility of simply parameterizing the query
commands in all cases as discussed, however, having gone down the
daggerization path and having gotten it to work, I have to say I find this
approach preferrable. There's really no case for identifying
cached/non-cached on a per-command basis and doing so would require
propagating the flag throughout all levels of the API and all callsites.
Tested: In addition to the new unit test which explicitly verifies the
caching/noncaching behavior of the new commands, tested the actual failing
sequence from "nomulus -e sandbox shell" and verified that the correct results
are served after a mutation.
* Fixed copyright year
* More copyright date fixes
* Added WhoisCommandFactoryTest to fragile tests
I suspect that this test has some contention with other tests, it's not clear
why.
* Update a few plugins for Java 11 compatibility
Guice 5.0.1 is now compatible with Java 11. However we don't
directly depend on Guice. Rather Soy depends on Guice. So I added a
direct dependency on Guice 5.0 just before Soy in order to frontload Soy
and pull in the newer version.
Mockito 3.7.7 is now compatible with Java 11. The complication is that
we need to use the inline version of Mockito, which among other things
also allows mocking for final classes (hooray!). It will eventually
become the default Mockito mock maker but for now it needs to be
manually activated.
Note that the inline version now introduces another warning:
```
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
```
Which I think is WAI due to how the inline mock maker works. Waiting on
the author to confirm.
After these to changes the only illegal reflective access is caused by
App Engine SDK tools, which we will rid ourselves of when we migrate off
of GAE.
* Restore package-lock.json
* Properly set up JPA in BEAM workers
Sets up a singleton JpaTransactionManger on each worker JVM for all
pipeline nodes to share.
Also added/updated relevant dependencies. The BEAM SDK version change
caused the InitSqlPipeline's graph to change.
* Update the Datastore to SQL migration pipeline
The pipeline now includes all entity types to be migrated by it, and has
completed successfully using the Sandbox data set. The running time in Sandbox
is about 3 hours, extrapolating by entity count to a 12-hour run with
production data. However, actual running time is likely to be longer since
throughput is lower with domains, which accounts for a higher percentage
of the total in production. More optimization will be needed.
The migrated data has not been validated.
Without it we kept getting the following warning:
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
* Fix JPA setup in Nomulus tool
Hibernate unnecessarily scans third-party classes in the Nomulus tool,
hitting a bug and fails to set up.
In this change we properly configured persistence.xml to include the orm mapping file (orm.xml) and disable
auto detection, and provided a custom (NOOP) scanner
to work around Hibernate scanner bugs.
Also improved on the :core:registryIntegrationTest task to test for
JPA setup as well as dependency-packaging.
For some inexplicable reasons I have to move the javax.mail package one
spot up to avoid its classes being shadowed by those provided in the
appengine package...
* Add use of interval data type
* Add support for Millis
* Use Java-object type
* Change column type for relock_duration
* add years and months
* Add tests for hours, minutes, and seconds
* Add javadoc describing how joda duration is stored
* Add test for lots of days
It is not clear to me why this folder is deleted in standardTest first,
presumably this is done when the webdriver tests were part of
standardTest?
Also not clear to me is why screenshots are only deleted in presubmits
but not locally. At least on my workstation standardTest runs after
fragileTest, just like in kokoro, so it should have deleted the
screenshots
folder, but it didn't.
Regardless, the correct thing to do is to delete this folder first in
fragileTest so that it doesn't interfere with the results.
Add the class paths of the source files generated by annotation processors to
the javadoc task's class path so that it doesn't complain about missing
Dagger classes.
Also remove empty <p> tags in all generated source files, because jaxb
genrerates files in multiple locations.
Lastly, for unkown reasons when the source level is set to > 8, the core
subproject throws a warning about a Gradle internal annotator processor
that only supports up to Java 8 and cause the Java compliation to fail because
we set -Werror on all java compliation tasks. I don't think there is a
strong reason that we set -Werror anyway, so this commit removes it.
1. The Gradle apt plugin is no longer needed to process annotations.
2. Without the apt plugin, Gralde puts the source files generated by
annotation processors in build/generated/sources/annotationProcessor.
3. Change the location of custom generated files to be consistent.
4. Fix a javadoc formatting error.
* Run InitSqlPipeline
Added the main() method to InitSqlPipeline.
Added a Gradle task to run InitSqlPipeline from command line. This
task is meant for testing and experiments.
Corrected the file name prefix of Datastore export files. Should
be 'output-', defined as 'input-'.
This makes it easier to later migrate the package to Java 11. If we move
and migrate in a single PR, because of the portion of the contents that
s changed, git will have trouble recognizing that some files are
renamed *and* modified and treat them as distinct files, making code
review difficult.
* Upgrade App Engine and webserver tests from JUnit 4 to 5
* Fix most errors
* Merge branch 'master' into junit5ification
* Fix test server by extracting non-test setup/tear-down
* Merge branch 'master' into junit5ification
* Fix backup tests
* Don't createFile(); asCharSink does it
* Increase the timeout for all WebDriver tests to 60s (helps w/ flakiness)
* End-to-end Datastore to SQL pipeline
Defined InitSqlPipeline that performs end-to-end migration from
a Datastore backup to a SQL database.
Also fixed/refined multiple tests related to this migration.
* Add JUnit Params and start using it
* Convert rest of RDE tests
* Don't check headers for generated tests
* Expand visibility to fix build breakage
* Bump JUnit versions to 5.6.2
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.
ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.
Added a mockito dependency, and upgraded mockito-core to 3.3.3.
Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
* Add testcontainers' Junit5 support dependency
Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.
Migrated PersistenceModuleTest as an example.
Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
On certain machines (one of mine) the outcastTest consistently fails due
to the following error:
java.lang.NoClassDefFoundError: Could not initialize class
google.registry.persistence.transaction.JpaTestRules$JpaIntegrationTestRule
If I reduce maxparallelForks to 3 it consistently passes. This issue was
mentioned here:
https://discuss.gradle.org/t/junit-test-fails-with-noclassdeffounderror-only-when-maxparallelforks-1/6047
But this post was 8 years old and no solution was identified.
* Hack to call setup and teardown in JUnit5 suite
JUnit 5 runner does not support @BeforeAll and @AfterAll declared
in the Suite class (as opposed to the member classes). However,
staying with the JUnit 4 suite runner would prevent any member
classes from migrating to JUnit 5.
We use a hack to invoke suite-level set up and teardown from tests.
This change is safe in that if the JUnit 5 runner implementation changes
behavior, we will only see false alarms.
* Exclude proxy configs from the FOSS jar
No sensitve data exposed.
Added a todo to modify the release process and stop
building the foss jar on the merged repo.
* Add Test suite support for JUnit 5 classes
Added Gradle dependencies and updated lockfiles.
Updated SqlInegrationTestSuite to use new annotations.
Migrated one member class in SqlIntegrationTestSuite (CursorDaoTest)
to JUnit 5, and verified that the new Suite runner can handle a
mixture of JUnit 4 and 5 tests in one suite.
Note that Gradle tests that run TestSuites must choose JUnit 4.
Updated core/build.gradle and integration/build.gradle.
* Actually run JUnit 5 tests
In Gradle, JUnit 5 must be explicitly enabled with a call to
test.useJUnitPlatform().
The FilteringTest used in :core must also enable JUnit5 separately.
Fixed AppEngineRule to work with the few tests that have migrated
to JUnit5.
More work is needed with AppEngine before we can migrate tests
that actually use Cloud SQL.
For context, with @EnableRuleMigrationSupport, JUnit 5 runner calls
an external resource's before() and after() methods. TestRule.apply()
is not called, therefore any setup done their will be bypassed with
JUnit 5.
* Allow backwards compatibility with JUnit 4 @Rules in JUnit 5
This allows us to defer having to re-implement all of our JUnit 4 Rules as JUnit
5 extensions for now, while continuing to in-place upgrade all existing JUnit 4
test classes to JUnit 5.
As proof of concept, this upgrades PremiumListUtils (which uses AppEngineRule,
our largest and most complicated @Rule) to use the JUnit 5 test runner.
* Apply formatter to entire file
* Make jpaTm for nomulus tool use local credential
* Remove unused methods in RegistryToolEnvironment
* Fix order of annotations
* Remove unused method in PersistenceComponent
* Move the creation of credential to the module
* Move creadential creation to AuthModule
* Add a TODO
* Enable Cloud SQL when Datastore is enabled for unit test
* Add explanation for why add a ETA field in GenerateEscrowDepositCommand
* Fix line length
* Ignore membershipt test but bring back test suite
* Fix tiny issue
* Fix broken builds when Maven Central is used
Gradle 6.2.1 apparently introduces a behavior change wrt boolean
expression: empty string used to eval to false, but now evals to
true.
Pre Gradle 6.2.1, root project's Gradle properties apparently were
not set to buildSrc. Now they are passed on to buildSrc -- mavenUrl
in buildSrc changes from null to "".
Both changes break the project when mavenUrl and/or pluginsUrl are
not set on command line.
Also added junit.jupiter-api as testCompile dependencies to projects.
This is a directly used dependency, whose absence causes a Lint
warning.
* Start using JUnit 5
This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration
In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.
* Delete no longer needed lockfiles
* Merge branch 'master' into first-junit5
* Implement dump_golden_schema command in devtool
Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.
Refactor the SQL container functionality from GenerateSqlCommand. There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.
* Reformatted and changes in response to review
* Fix getDockerTag() usage
* Fix "leaked resource"
