* Fix empty Domain nameserver loads
Domains with no nameservers were being loaded from SQL as an empty set instead
of null as they should be.
Discovered this will trying to test updates, so added a test for updates in
the course of it.
* Add a SQL schema to AllocationToken
* Respond to CR
- rename field in tests
- rename allowed_registrar_ids field
- remove unnecessary db load in GATC
* Add TODO for HistoryEntry vkeys
* Run autoformat
* V48 -> V49
* Allow schema push in alpha and crash
This allows unsubmitted changes to be tested in alpha and
crash.
Also updated the README file and reformatted with the internal
mdformat tool.
* Update user-facing documentation
Give our docs a complete overhaul to account for changes in the system,
notably the requirement to configure postgresql.
* Fix dangling sentence.
* Merge branch 'master' into admin-docs
For some inexplicable reasons I have to move the javax.mail package one
spot up to avoid its classes being shadowed by those provided in the
appengine package...
* Create DAO for Spec11ThreatMatch
* Add tests
* Execute SQL for deleteEntriesByDategit status
* Remove testing line
* Rename createSpec11ThreatMatch()
* Add comments about jpaTm and use jpaTm() in test
* Fix technicality in comment
* Remove a new line
* Truth chaining for comparing ImmutableLists of matches
* Javadoc formatting
* Add some missing @Nullables and types
Also deletes two unused VKey.createOfy() methods that simply don't work, because
a kind and an id is not enough to create a Datastore key; you also need the full
entity group inheritance chain for entities that are not roots themselves
(which is most of the entities in our schema).
* Merge branch 'master' into add-missing-nullables
* Throw UnsupportedOperationException for contacts/hosts too
* Merge branch 'master' into add-missing-nullables
* Update BEAM SDK to work with Java 11
Upgraded BEAM dependencies to 2.23.0.
Updated Spec11 and invoice pipelines:
- Added the required region parameter.
- Removed the workaround code for staging.
Verified that staging is successful in alpha:
./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_spec11_pipeline --project domain-registry-alpha'
and
./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_invoicing_pipeline'
* Double the timeout length for WebDriver screenshot tests
My theory is that this timeout is being applied to all retries of a failing test
rather than each one, and thus flaky screenshot tests aren't being given
sufficient time to complete any attempt past the first.
This test is flaky and the going theory is that we take a screenshot
immediately after we click the "Add" button and the CSS hover effect is
somehow still playing on the button.
* Restore original jpa manager after tests
AppEngineExtensionTest fails to restore original jpa manager in some
tests. This results in flakiness in DummyJpaTransactionManagerTest.
The current setup causes the GCB job to fail validation and not run because it
uses backticks in the configuration yaml, which is not allowed -- there is no
shell to perform backtick substitution. See the error message here:
https://spinnaker.endpoints.domain-registry-dev.cloud.goog/gate/pipelines/01EF5GRMD625613H6Z033DBD3Z
In the future please make sure to test the GCB pipeline as instructed in
the comments at the beginning of each file before committing.
I tried to work around it by downloading the nomulus tool jar file
instead (running the nomulus-tool docker image inside a docker image is
not advisable). However the "nomulus deploy_spec11_pipeline" command
still fails. I'm not sure why. Has the command itself been tested
locally? The error message is shown below:
```
Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
Step #2: WARNING: --region not set; will default to us-central1. Future releases of Beam will require the user to set the region explicitly. https://cloud.google.com/compute/docs/regions-zones/regions-zones
Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory tryCreateDefaultBucket
Step #2: INFO: No tempLocation specified, attempting to use default bucket: dataflow-staging-us-central1-937378958468
Step #2: Aug 09, 2020 3:11:47 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse
Step #2: WARNING: Request failed with code 409, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://www.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=domain-registry-alpha.
Step #2: Exception in thread "main"
Step #2: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
Step #2: at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
Step #2:
Step #2: at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
Step #2: at org.apache.beam.sdk.Pipeline.create(Pipeline.java:147)
Step #2:
Step #2: at google.registry.beam.spec11.Spec11Pipeline.deploy(Spec11Pipeline.java:157)
Step #2: at google.registry.tools.DeploySpec11PipelineCommand.run(DeploySpec11PipelineCommand.java:80)
Step #2: at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:257)
Step #2: at google.registry.tools.RegistryCli.run(RegistryCli.java:182)
Step #2: at google.registry.tools.RegistryTool.main(RegistryTool.java:129)
Step #2: Caused by: java.lang.reflect.InvocationTargetException
Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Step #2:
Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Step #2: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Step #2: at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
Step #2: ... 8 more
Step #2: Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@5cb0d902, only URLClassLoaders are supported.
Step #2: at org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:58)
Step #2:
Step #2: at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:285)
Step #2:
Step #2: ... 13 more
```
Lastly the "--project" flag refers to the KMS project. While I'm not
sure which project is that, I don't think we can use the PROJECT_ID
variable as this is a GCB-substituted variable which refers to the
project that the GCB job runs in, which in our cases means
domain-registry-dev. We shouldn't use that project for KMS. I've changed
it to the same project as the one we are deploying to, but please note
that we have a separate project ${project_id}-keys that is used for all
KMS purposes. This is specified in the config file so if that's what you
meant to use, there is no need to specify it in the command line. Actually
if you meant use the project to be deployed to for KMS, it also
shouldn't be necessary to specify it separately as this information is
already known when you specified "nomulus -e ENV".
https://team.git.corp.google.com/domain-registry-eng/nomulus-internal/+/refs/heads/master/core/src/main/java/google/registry/config/files/nomulus-config-production.yaml#168
Can you add more description on what the KMS project is supposed to be?
I don't think we specify a project for KMS purpose in any other
commands.
Given that there are several unresolved issues, I've commented out my
proposed solution so that deployment can proceed.
For reasons unclear at the moment the tests are not passing. Disabling
them for now so that release candidates can be built. We have CI runs
after each merge so we should be pretty confident if the build is broken
or not.
The node installed by nvm gives errors when running "npm install".
Also installs Python as it is need. Presumbly the system provided npm
version has python as a dependency so it was installed when npm was
installed.
* Add retrier to retry exceptions we've seen so far
* Check for nested exceptions
* fix formatting
* check for nested transactions
* add JDBCConnectionException to isFailedQueryRetriable
* Add retrier to methods with suppliers
- Reuse DS record format processing from the create/update domain commands
(BIND format, commonly used in URS requests)
- Remove the CLIENT_HOLD status from domains that have it (this blocks us from
serving the new nameservers and DS record)
* Replace jpaTm with a JpaSupplierFactory
* Style
* Style
* Pipeline takes in a SerializableSupplier instead
* Change the ordering of imports
* Test a good domain in addition to a bad one
* Rename and check good domain for Transact Answer
* Use standard Mockito verify
* Verify transact call and no more interactions
* Remove Answer comment
* Naming chsnges
* Deploy Spec 11 pipeline correctly
* Fix formatting of deploy file
* Use a file to persist state across Cloud Build steps
Co-authored-by: Gus Brodman <gbrodman@google.com>
* Refactor DomainBase into DomainContent and create DomainHistory
This is similar to #587 and #634, but for domains.
One caveat is that we refactor some of the Domain* instance methods to
be static so that they can be called either on DomainBase or
DomainContent, returning the appropriate type each time.
Note that we set DomainHistory to use the same revision ID sequence as
HostHistory and ContactHistory.
In addition, we refactor the tests to the History objects a bit to
reduce duplicate code and because we cannot guarantee yet that the
SQL-stored VKeys are symmetrical -- the ofy keys are not persisted at
the moment.
In addition, rename the DomainHost table to the default Domain_nsHosts so that it automatically creates two separate nsHosts tables for us -- one foreign-keyed on the domain repo ID, and one foreign-keyed on the history revision ID
* Use access hackery to allow manual names for nsHosts tables
* Clean up post merge artifacts
* Add unused setters that Hibernate requires
* Fix the tests and semantic merge conflicts
* Change ns_hosts to ns_host everywhere
* Rename ns_host to host_repo_id
* V42 -> V44
* Enable Java 11 features
As of this commit Java 11 must be used to build. The generated bytecode
is still at Java 8 due to App Engine task queue limit.
Also fixed a bug where the included google-java-format jar file is not
used, requiring the user to install it separately.
See: https://cloud.google.com/appengine/docs/standard/java/taskqueue/push
* Add use of interval data type
* Add support for Millis
* Use Java-object type
* Change column type for relock_duration
* add years and months
* Add tests for hours, minutes, and seconds
* Add javadoc describing how joda duration is stored
* Add test for lots of days
It is not clear to me why this folder is deleted in standardTest first,
presumably this is done when the webdriver tests were part of
standardTest?
Also not clear to me is why screenshots are only deleted in presubmits
but not locally. At least on my workstation standardTest runs after
fragileTest, just like in kokoro, so it should have deleted the
screenshots
folder, but it didn't.
Regardless, the correct thing to do is to delete this folder first in
fragileTest so that it doesn't interfere with the results.
Add the class paths of the source files generated by annotation processors to
the javadoc task's class path so that it doesn't complain about missing
Dagger classes.
Also remove empty <p> tags in all generated source files, because jaxb
genrerates files in multiple locations.
Lastly, for unkown reasons when the source level is set to > 8, the core
subproject throws a warning about a Gradle internal annotator processor
that only supports up to Java 8 and cause the Java compliation to fail because
we set -Werror on all java compliation tasks. I don't think there is a
strong reason that we set -Werror anyway, so this commit removes it.
* Get rid of all remaining JUnit 4 usages except in prober & proxy subprojects
Caveat: Test suites aren't yet implemented in JUnit 5 so we still use the ones
from JUnit 5 in the core subproject.
* Fix some build errors
* Migrate the documentation package to Java 11
The old Doclet API is deprected and removed in Java 12. This commit
changes the documentation package to use the new recommended API.
However it is not a drop-in replacement and there are non-idiomatic
usages all over the place. I think it is eaiser to keep the current code
logic and kind of shoehorn in the new API than starting afresh as the
return on investment of a do-over is not great.
Also note that the docs package is disabled as of this commit because we
are still using Java 8 to compile which lacks the new API. Once we
switch our toolchains to Java 11 (but still compiling Java 8 bytecode)
we can re-enable this package.
TESTED=ran `./gradlew :docs:test` locally with the documentation package
enabled.
1. The Gradle apt plugin is no longer needed to process annotations.
2. Without the apt plugin, Gralde puts the source files generated by
annotation processors in build/generated/sources/annotationProcessor.
3. Change the location of custom generated files to be consistent.
4. Fix a javadoc formatting error.
* Run InitSqlPipeline
Added the main() method to InitSqlPipeline.
Added a Gradle task to run InitSqlPipeline from command line. This
task is meant for testing and experiments.
Corrected the file name prefix of Datastore export files. Should
be 'output-', defined as 'input-'.
Apparently, in domain check responses, `avail=false, reason=Allocation token
required` was not sufficiently understood by all registrars. This changes it to
`avail=false, reason=Reserved; alloc. token required` to hopefully make it
crystal clear that the domain in question is reserved, i.e. if you were supposed
to be able to register this domain you'd already know it because we'd have
already given you the requisite allocation token.
