Commit graph

3180 commits

Author SHA1 Message Date
Lai Jiang
bfd61ef867
Fix another bug in the proxy (#419)
The promise should be set outside the try block because if we want
warning only, we still want the promise to be set even if the
clientCertificate.checkValidity() throws an error.
2019-12-18 16:24:23 -05:00
Weimin Yu
ffe3eb1548
Remove the schema jar url override for tests (#415)
* Remove the schema jar url override for tests

Revert pull request 374: "Allow schema-loading from arbitrary url in tests".

Pull request 403 (f48e3933f5) is a more
general solution.

This reverts commit 68887d427f.
2019-12-17 11:24:20 -05:00
Weimin Yu
6aaf081489
Create a new app to hold GenerateSqlSchemaCommand (#409)
* Create a new app to hold GenerateSqlSchemaCommand

GenerateSqlSchemaCommand starts postgresql using testcontainer.
This makes junit etc a runtime dependency, allowing them to get
into release artifacts.

By moving this command to a separate tool, we can remove junit
etc as compile/runtime dependency.
2019-12-13 16:05:35 -05:00
Weimin Yu
1143b25391
Fix test on use of external schema jar (#412)
* Fix test on use of external schema jar

One test breaks if path to external schema is set and
test is run from the sqlIntegrationTest task.
2019-12-13 11:32:19 -05:00
Weimin Yu
65cf49f204
Fix sql script name conflict (#411)
* Fix sql script name conflict

There are two V11__ files due to concurrent merge. Renamed one
to V12__

Also removed a @NotNull annotation, which is the fist in the code base.
Most of the code base use @Nullable instead. If we do want to use
@NotNull, we may want to use the javax one instead.
2019-12-12 16:16:43 -05:00
Weimin Yu
f48e3933f5
Run cross-release SQL integration tests (#403)
* Run cross-release SQL integration tests

Run SQL integration tests across arbitrary schema and server
releases.

Refer to integration/README.md in this change for more information.

TESTED=Cloud build changes tested with cloud-build-local
       Used the published jars to test sqlIntegration task locally.
2019-12-12 13:47:49 -05:00
Lai Jiang
9853f23d94
Fix null pointer excpetion bug (#407)
The factory method passes a null trustedCertificates instead of an empty
list.
2019-12-12 13:06:43 -05:00
Ben McIlwain
db7fcf6c38
Add Cloud SQL premium list caches and compare prices with Datastore (#376)
* Add Cloud SQL premium list caches and compare prices with Datastore

Nothing will fail if the prices can't be loaded from Cloud SQL, or if the prices
are different. All that happens is that the error is logged. Then, once this is
running in production for awhile, we'll look at the logs and see if there will
be any pricing implications from switching over to the Cloud SQL version of the
premium lists.

* Add setMaxResults(1) per code review

* Add tests and reorder public functions

* Don't statically import caches

* Improve test pass rate

* Merge branch 'master' into dual-read-premium

* Add PremiumEntry mapping

* Allow update

* Revert column order

* Alphabetize PremiumEntry columns

* Don't bother trying to enforce order

* Private constructor
2019-12-11 16:20:19 -05:00
Weimin Yu
3aad8b6aa7
Use TextDiffSubject to compare multi-line text (#406)
* Use TextDiffSubject to compare  multi-line text

It illustrates differences better.

Moved TextDiffSubject.java to the common project for sharing.
2019-12-10 16:44:59 -05:00
Weimin Yu
f7b243e390
Make devProject a project property (#405)
* Make devProject a project property

Properties set in rootProject's ext block are not overridden by
commandline flags.

* Make flyway commands runnable from FOSS repo

rootProject.devProject, defined in projects.gradle, cannot be
overridden by commandline flags. Added a flag check in :db's
project script.
2019-12-10 11:39:56 -05:00
Michael Muller
e8745f7538
Update lockfiles (#402)
* Update lockfiles

Update lockfiles for the latest dependencies and metadata.
2019-12-10 09:28:44 -05:00
sarahcaseybot
2478a4a93b
Add schema and DAO for cursors in cloudsql (#370)
* Add schema for Cursor

* Add CursorDao and CursorDaoTest

* Fix comment on getTld

* Change tld column to scope

* Fix cursorTime to be converted to DateTime internally and other small fixes

* Add a CursorType enum and a createGlobal constructor for Cursor

* Rename flyway file

* Use cursorType from common/Cursor.java and add null checks
2019-12-09 17:47:06 -05:00
Weimin Yu
bba5aff4b6
Verify RegistryTool can instantiate (#400)
* Verify RegistryTool can instantiate

Add a task that instantiates all command classes in RegistryTool
with runtimeClasspath.

Also make sure that runtimeClasspath is a superset of
compileClasspath.
2019-12-06 12:08:16 -05:00
Michael Muller
df7d272157
Adding junit back into the runtime classpath (#399)
* Adding junit back into the runtime classpath

Unfortunately, GenerateSqlSchemaCommand depends on junit via testcontainers.
We should really move GenerateSqlSchemaCommand out of nomulus tool (we only
use it during development) but this gets nomulus tool working for the time
being.

* Removed unnnecessary trace line.

* lockfiles generated after update_dependency.sh
2019-12-05 16:01:05 -05:00
Shicong Huang
988f78274e
Check SQL dependency in super class for SqlIntegrationMembershipTest (#398)
In some cases, we define JpaTransactionManagerRule in a TestCase
class which is extended by the concrete test class. So, we need
to check if JpaTransactionManagerRule is also defined in the super
class.
2019-12-05 13:12:56 -05:00
Lai Jiang
369c1259fb
Remove AUTH_INTERNAL_ONLY (#393)
Endpoints annotated with AUTH_INTERNAL_ONLY used to be accessible
manually with an internal RPC tool that adds App Engine specific HTTP
headers to a request to make it look like it comes from App Engine
(hence internal). This tool is used by admins to hit such endpoints
during debugging, making them effectively AUTH_INTERNAL_OR_ADMIN.

This RPC tool has never been made available outside Google so the open
source admins do not have such ability. A recent change in the RPC tool
made this hack stop working internally as well. This PR replaces all
all occurances of AUTH_INTERNAL_ONLY with AUTH_INTERNAL_OR_ADMIN and
brings the open source build into feature parity with the internal
version.

Also fixed a few issues the router tests.
2019-12-04 12:52:18 -05:00
Michael Muller
1e1c8cdd80
Replace TransactionManager.Work with Supplier (#397)
* Replace TransactionManager.Work with Supplier

Replace the "Work" class with the equivalent java.util.function.Supplier.
2019-12-04 12:00:13 -05:00
Weimin Yu
3d3d390a9f
Fix bad dependency lock file (#396)
* Fix bad dependency lock file
2019-12-03 13:31:57 -05:00
Weimin Yu
f0fc53e035
Allow project dependency to use runtimeClasspath (#395)
* Allow project dependency to use runtimeClasspath

Project dependency should use runtimeClasspath. However, if
left unspecified, it uses 'default', which is the same as
the legacy 'runtime' configuration. (runtimeOnly dependencies
are left out).

Since runtimeClasspath cannot be referenced directly, we use
a custom config (deploy_jar) as a proxy.

By excluding testjars (leaked into 'compile' by third-party
dependencies) from runtimeClasspath, we prevent them from
getting into release artifacts.

Two meaningful changes in appengine_war.gradle and java_common.gradle

TESTED=Diffed contents of services/{module}/build/exploded-*
       Only three jars are removed: hamcrest-core, junit, and
       mockito-core.
2019-12-02 16:10:13 -05:00
Lai Jiang
c156842844
Upgrade to Gradle 6.0.1 (#394) 2019-12-02 15:24:18 -05:00
Lai Jiang
c584de9f72
Respect certificate validity period (#391)
Client SSL handler already performs the necessary validation. Only tests are
added.

Server SSL handler does not currently check for the validity period of
the client certificate as the insecure trust manager is used. This PR
added the check but does not actually terminate the connection yet. It
will log the expired certificates so that we can contact the registrars
to update them.

Once we are certain that all certificates are updated, we can turn off
dryrun mode.

We should also consider checking if the certificate has too long a
validity period as it defeats the purpose of using regularly updated
certificates to deprecate insecure cipher suites.
2019-11-27 16:08:38 -05:00
Shicong Huang
9be5091c84
Add entity for reserved list (#381)
This PR added the Cloud SQL entity for reserved list.
2019-11-26 16:51:41 -05:00
Michael Muller
28499d23a0
Print filenames that need to be reformatted (#386)
* Print filenames that need to be reformatted

Print the names of all java files that need reformatting during the check and
reformat operations.
2019-11-26 13:20:27 -05:00
Ben McIlwain
961d7e88f4
Use Maps.transformEntries() utility method to improve Map composition (#387)
* Use Maps.transformEntries() utility method to improve Map composition
2019-11-26 12:20:00 -05:00
Weimin Yu
215de62fa7
Stop publish Cloud SQL schema jar to maven repo (#383)
* Stop publish Cloud SQL schema jar to maven repo

The original purpose of the maven publication is for
use in server/schema compatibility tests. A commandline
flag can direct a test run to use different versions of
the schema jar. However, this won't work due to dependency
locking.
2019-11-25 18:23:02 -05:00
Lai Jiang
05d56fe1a2
Remove SSL initializer from the prober (#378)
The prober now uses the common SSL initializer in the networking
subproject.

Also changed both initializers to take an ImmutableList of certificates
other than an array of those, for better immutability.

I have no idea where these lockfile changes are coming from. They seem
to be pure noise as far as code review is concerned.
2019-11-22 17:46:06 -05:00
sarahcaseybot
e318f47fc6
Add a cursor for tracking monthly uploads of ICANN report (#343)
* Add a cursor for tracking monthly uploads of the transaction report to ICANN

* Add cursors to track activity, transaction, and manifest report uploads.

* Address comments

* Add @Nullable annotation to manifestCursor

* Add lock and batch load cursors.

* Add string formatting, autovalue CursorInfo object, and handling for null cursors

* Add some helper functions for loadCursors and restructure to require less round trips to the database

* Switch new cursors to be created with cursorTime at first of next month
2019-11-22 17:40:31 -05:00
Lai Jiang
cc5f62587e
Make dev project configurable (#371)
* Make dev project configurable

We should not hardcode our dev project in the public config file.

* Remove the use of .ext when using external properties

They are only needed when defining properties.
2019-11-22 16:20:07 -05:00
Lai Jiang
02846bcbdd
No-op: Use nicer HCL2 syntax. (#384)
Generated with perl -pi -e 's/\"\$\{([a-zA-Z0-9._-]*)\}\"/$1/g' $(find ./ -name '*.tf')

Copied from cl/282012376.
2019-11-22 16:08:56 -05:00
Ben McIlwain
c920f709ef
Update the Registries cache to leverage/populate the Registry cache (#382)
* Update the Registries cache to leverage/populate the Registry cache

This is accomplished by also providing a loadAll() method on the Registry cache
that can be used to load an entire batch of Registry objects at once.

This improves efficiency, because now, any operation on Registries that loads
all the Registry entities (getTlds(), getTldsOfType(), and getTldEntities()), is
plumbed through the Registry cache, therefore loading it from that cache if it
exists and only hitting the DB if not. If not, this populates the Registry cache
upon loading, so that subsequent calls to Registry.get() will now hit the cache
instead of the DB.

To give a concrete example, the following code:

    for (String tld : Registries.getTlds()) {
      // ...
      doSomethingWith(Registry.get(tld));
      // ...
    }

is now much more efficient, because the initial call to Registries.getTlds()
populates all the entities in cache, and the subsequent individual calls to
Registry.get(tld) now retrieve them from the cache. Prior to this change,
Registries.getTlds() did not populate the Registry cache, and each subsequent
Registry.get(tld) had the potential to trigger an individual round-trip to the
DB, which is obviously bad for performance.
2019-11-22 14:47:09 -05:00
Ben McIlwain
ad9daac1ab
Update premium and reserved list management docs (#380)
* Update premium and reserved list management docs

They were a little bit out of date.
2019-11-21 16:44:45 -05:00
Weimin Yu
9f0e24132a
Break circular dependency between core and util (#379)
* Break circular dependency between core and util

Created a new :common project and moved a minimum
number of classes to break the circular dependency
between the two projects. This gets rid of the
gradle lint dependency warnings.

Also separated api classes and testing helpers into
separate source sets in :common so that testing
classes may be restricted to test configurations.
2019-11-21 15:36:55 -05:00
Shicong Huang
98414cb7cb
Add a test to verify generated schema (#377) 2019-11-21 13:37:37 -05:00
Lai Jiang
6af1896362
Refactor common code used by the proxy and the prober (#375) 2019-11-20 12:42:44 -05:00
Weimin Yu
68887d427f
Allow schema-loading from arbitrary url in tests (#374)
* Allow schema-loading from arbitrary url in tests

Server/Schema compatibility tests must be able to load different versions
of the SQL schema. This change allows test runners to override the
schema location using a system property.

Note: due to dependency-locking, we cannot manipulate the dependencies
closure in the build script to load different schema jars. The jars
must not be on the classpath.
2019-11-20 12:22:48 -05:00
Ben McIlwain
8a06ef09c0
Add a new method to load all Registry entities of a given type (#373)
* Add a new method to load all Registry entities of a given type

This is useful for things that need to know more than just the TLD strings
themselves, which is all Registries currently provides.
2019-11-18 17:40:42 -05:00
Shicong Huang
9e0368b77c
Enable JpaTransactionManager in all environment (#352)
* Enable JpaTransactionManager in all environment

* Refactor to have a single place to create JpaTm
2019-11-18 14:53:49 -05:00
Weimin Yu
05c45da07a
Use psql 11 docker image in all tests (#372)
* Use psql 11 docker image in all tests
2019-11-18 14:08:58 -05:00
Weimin Yu
365c5da942
Require explict tag when starting psql docker (#368)
* Require explict tag when starting psql docker

Defined a util class to return docker tag of desired PSQL version.
Class is defined in ':db' and shared by ':db' and ':core'. Used
an artifact declaration to exclude unnecesary compile dependencies.

Added a presubmit check for instantiations without explicit tag.
2019-11-18 11:33:26 -05:00
Weimin Yu
2cc2571375
Update schema deployment doc and flyway tool (#363)
* Update schema deployment doc and flyway tool

Disabled Flyway Gradle tasks with side effects on Cloud SQL
instances. They can still be used on local databases.

Also switched Flyway Gradle tasks to get credentials from
new locations (in domain-registry-dev).

Updated README file on schema push process. Also reformatted
the entire file.
2019-11-15 11:44:21 -05:00
Lai Jiang
d55230933b
Upgrade to Gradle 6.0 (#364) 2019-11-13 13:17:29 -05:00
Shicong Huang
09aef04117
Add PersistenceXmlUtility and refactor related code (#357)
* Refactor GenerateSqlSchemaCommand

* Add and throw UncheckedClassNotFoundException
2019-11-12 15:08:54 -05:00
gbrodman
a392100852
Add by-repo-id method to RegistryLockDao (#344)
* Add by-repo-id method to RegistryLockDao

When we create new registry lock objects on user request, we will want
to make sure that there are no pending lock/unlock actions on this
domain. In order to do that, we will need to know what lock actions
there have been for this domain.

* Inline the query into a single statement

* whoops

* comments
2019-11-12 11:47:59 -05:00
Weimin Yu
b005e3aeb0
Fix a few lint errors (#361)
Replace deprecated bouncycastle class in SslInitializerTestUils.

Generic array as vargs: worked around it in ProbingAction and
removed unused method in CircularList.
2019-11-12 11:14:51 -05:00
Weimin Yu
bce09a3aa3
Deploy SQL schema from Cloud Build (#350)
Defined Docker image for schema deployment.

Included schema deploymer docker in the Cloud Build release process.

Defined Cloud Build config for schema deployment.

TESTED=Used cloud-build-local to test deployment flow
TESTED=Used docker to test schema deployer image in more ways
2019-11-12 11:06:16 -05:00
Weimin Yu
cb87658827
Remove unnecessary usage of CircularList (#358)
Implementation of CircularList is an overkill for its stated
purpose. Remove an unnecessary usage so that the remaining
use case may be refactored in isolation.
2019-11-11 18:04:46 -05:00
Michael Muller
455daae25c
Add converter for DateTime (#346)
* Add converter for DateTime

* Added to sql integration test suite

* Removed obsolete "auto update schema" property
2019-11-11 17:24:20 -05:00
gbrodman
dea7dfcf28
Refactor UI actions to be more reliable (#348)
* Refactor UI actions to be more reliable

- Created HtmlAction to handle the nitty-gritty of login and setting up
HTML pages
- Created a test to verify that all UI actions implement JSON GET, JSON
POST, or HTML classes
- Move CSS renaming into a utility class

* Move logging of request into HtmlAction

* Comment and wording in exception

* mention JsonGetAction in the comment

* JsonGetAction extends Runnable
2019-11-11 16:18:48 -05:00
Shicong Huang
3a47fa2fe9
Fix beam pipeline deployment issue (#360) 2019-11-11 15:57:33 -05:00
gbrodman
0b7a0ba99d
Update puppeteer (#359)
* Update puppeteer

They released 2.0.0 which fixes the security vuln in https-proxy-agent
2019-11-11 15:38:44 -05:00