Commit graph

2966 commits

Author SHA1 Message Date
Shicong Huang
988f78274e
Check SQL dependency in super class for SqlIntegrationMembershipTest (#398)
In some cases, we define JpaTransactionManagerRule in a TestCase
class which is extended by the concrete test class. So, we need
to check if JpaTransactionManagerRule is also defined in the super
class.
2019-12-05 13:12:56 -05:00
Lai Jiang
369c1259fb
Remove AUTH_INTERNAL_ONLY (#393)
Endpoints annotated with AUTH_INTERNAL_ONLY used to be accessible
manually with an internal RPC tool that adds App Engine specific HTTP
headers to a request to make it look like it comes from App Engine
(hence internal). This tool is used by admins to hit such endpoints
during debugging, making them effectively AUTH_INTERNAL_OR_ADMIN.

This RPC tool has never been made available outside Google so the open
source admins do not have such ability. A recent change in the RPC tool
made this hack stop working internally as well. This PR replaces all
all occurances of AUTH_INTERNAL_ONLY with AUTH_INTERNAL_OR_ADMIN and
brings the open source build into feature parity with the internal
version.

Also fixed a few issues the router tests.
2019-12-04 12:52:18 -05:00
Michael Muller
1e1c8cdd80
Replace TransactionManager.Work with Supplier (#397)
* Replace TransactionManager.Work with Supplier

Replace the "Work" class with the equivalent java.util.function.Supplier.
2019-12-04 12:00:13 -05:00
Weimin Yu
3d3d390a9f
Fix bad dependency lock file (#396)
* Fix bad dependency lock file
2019-12-03 13:31:57 -05:00
Weimin Yu
f0fc53e035
Allow project dependency to use runtimeClasspath (#395)
* Allow project dependency to use runtimeClasspath

Project dependency should use runtimeClasspath. However, if
left unspecified, it uses 'default', which is the same as
the legacy 'runtime' configuration. (runtimeOnly dependencies
are left out).

Since runtimeClasspath cannot be referenced directly, we use
a custom config (deploy_jar) as a proxy.

By excluding testjars (leaked into 'compile' by third-party
dependencies) from runtimeClasspath, we prevent them from
getting into release artifacts.

Two meaningful changes in appengine_war.gradle and java_common.gradle

TESTED=Diffed contents of services/{module}/build/exploded-*
       Only three jars are removed: hamcrest-core, junit, and
       mockito-core.
2019-12-02 16:10:13 -05:00
Lai Jiang
c156842844
Upgrade to Gradle 6.0.1 (#394) 2019-12-02 15:24:18 -05:00
Lai Jiang
c584de9f72
Respect certificate validity period (#391)
Client SSL handler already performs the necessary validation. Only tests are
added.

Server SSL handler does not currently check for the validity period of
the client certificate as the insecure trust manager is used. This PR
added the check but does not actually terminate the connection yet. It
will log the expired certificates so that we can contact the registrars
to update them.

Once we are certain that all certificates are updated, we can turn off
dryrun mode.

We should also consider checking if the certificate has too long a
validity period as it defeats the purpose of using regularly updated
certificates to deprecate insecure cipher suites.
2019-11-27 16:08:38 -05:00
Shicong Huang
9be5091c84
Add entity for reserved list (#381)
This PR added the Cloud SQL entity for reserved list.
2019-11-26 16:51:41 -05:00
Michael Muller
28499d23a0
Print filenames that need to be reformatted (#386)
* Print filenames that need to be reformatted

Print the names of all java files that need reformatting during the check and
reformat operations.
2019-11-26 13:20:27 -05:00
Ben McIlwain
961d7e88f4
Use Maps.transformEntries() utility method to improve Map composition (#387)
* Use Maps.transformEntries() utility method to improve Map composition
2019-11-26 12:20:00 -05:00
Weimin Yu
215de62fa7
Stop publish Cloud SQL schema jar to maven repo (#383)
* Stop publish Cloud SQL schema jar to maven repo

The original purpose of the maven publication is for
use in server/schema compatibility tests. A commandline
flag can direct a test run to use different versions of
the schema jar. However, this won't work due to dependency
locking.
2019-11-25 18:23:02 -05:00
Lai Jiang
05d56fe1a2
Remove SSL initializer from the prober (#378)
The prober now uses the common SSL initializer in the networking
subproject.

Also changed both initializers to take an ImmutableList of certificates
other than an array of those, for better immutability.

I have no idea where these lockfile changes are coming from. They seem
to be pure noise as far as code review is concerned.
2019-11-22 17:46:06 -05:00
sarahcaseybot
e318f47fc6
Add a cursor for tracking monthly uploads of ICANN report (#343)
* Add a cursor for tracking monthly uploads of the transaction report to ICANN

* Add cursors to track activity, transaction, and manifest report uploads.

* Address comments

* Add @Nullable annotation to manifestCursor

* Add lock and batch load cursors.

* Add string formatting, autovalue CursorInfo object, and handling for null cursors

* Add some helper functions for loadCursors and restructure to require less round trips to the database

* Switch new cursors to be created with cursorTime at first of next month
2019-11-22 17:40:31 -05:00
Lai Jiang
cc5f62587e
Make dev project configurable (#371)
* Make dev project configurable

We should not hardcode our dev project in the public config file.

* Remove the use of .ext when using external properties

They are only needed when defining properties.
2019-11-22 16:20:07 -05:00
Lai Jiang
02846bcbdd
No-op: Use nicer HCL2 syntax. (#384)
Generated with perl -pi -e 's/\"\$\{([a-zA-Z0-9._-]*)\}\"/$1/g' $(find ./ -name '*.tf')

Copied from cl/282012376.
2019-11-22 16:08:56 -05:00
Ben McIlwain
c920f709ef
Update the Registries cache to leverage/populate the Registry cache (#382)
* Update the Registries cache to leverage/populate the Registry cache

This is accomplished by also providing a loadAll() method on the Registry cache
that can be used to load an entire batch of Registry objects at once.

This improves efficiency, because now, any operation on Registries that loads
all the Registry entities (getTlds(), getTldsOfType(), and getTldEntities()), is
plumbed through the Registry cache, therefore loading it from that cache if it
exists and only hitting the DB if not. If not, this populates the Registry cache
upon loading, so that subsequent calls to Registry.get() will now hit the cache
instead of the DB.

To give a concrete example, the following code:

    for (String tld : Registries.getTlds()) {
      // ...
      doSomethingWith(Registry.get(tld));
      // ...
    }

is now much more efficient, because the initial call to Registries.getTlds()
populates all the entities in cache, and the subsequent individual calls to
Registry.get(tld) now retrieve them from the cache. Prior to this change,
Registries.getTlds() did not populate the Registry cache, and each subsequent
Registry.get(tld) had the potential to trigger an individual round-trip to the
DB, which is obviously bad for performance.
2019-11-22 14:47:09 -05:00
Ben McIlwain
ad9daac1ab
Update premium and reserved list management docs (#380)
* Update premium and reserved list management docs

They were a little bit out of date.
2019-11-21 16:44:45 -05:00
Weimin Yu
9f0e24132a
Break circular dependency between core and util (#379)
* Break circular dependency between core and util

Created a new :common project and moved a minimum
number of classes to break the circular dependency
between the two projects. This gets rid of the
gradle lint dependency warnings.

Also separated api classes and testing helpers into
separate source sets in :common so that testing
classes may be restricted to test configurations.
2019-11-21 15:36:55 -05:00
Shicong Huang
98414cb7cb
Add a test to verify generated schema (#377) 2019-11-21 13:37:37 -05:00
Lai Jiang
6af1896362
Refactor common code used by the proxy and the prober (#375) 2019-11-20 12:42:44 -05:00
Weimin Yu
68887d427f
Allow schema-loading from arbitrary url in tests (#374)
* Allow schema-loading from arbitrary url in tests

Server/Schema compatibility tests must be able to load different versions
of the SQL schema. This change allows test runners to override the
schema location using a system property.

Note: due to dependency-locking, we cannot manipulate the dependencies
closure in the build script to load different schema jars. The jars
must not be on the classpath.
2019-11-20 12:22:48 -05:00
Ben McIlwain
8a06ef09c0
Add a new method to load all Registry entities of a given type (#373)
* Add a new method to load all Registry entities of a given type

This is useful for things that need to know more than just the TLD strings
themselves, which is all Registries currently provides.
2019-11-18 17:40:42 -05:00
Shicong Huang
9e0368b77c
Enable JpaTransactionManager in all environment (#352)
* Enable JpaTransactionManager in all environment

* Refactor to have a single place to create JpaTm
2019-11-18 14:53:49 -05:00
Weimin Yu
05c45da07a
Use psql 11 docker image in all tests (#372)
* Use psql 11 docker image in all tests
2019-11-18 14:08:58 -05:00
Weimin Yu
365c5da942
Require explict tag when starting psql docker (#368)
* Require explict tag when starting psql docker

Defined a util class to return docker tag of desired PSQL version.
Class is defined in ':db' and shared by ':db' and ':core'. Used
an artifact declaration to exclude unnecesary compile dependencies.

Added a presubmit check for instantiations without explicit tag.
2019-11-18 11:33:26 -05:00
Weimin Yu
2cc2571375
Update schema deployment doc and flyway tool (#363)
* Update schema deployment doc and flyway tool

Disabled Flyway Gradle tasks with side effects on Cloud SQL
instances. They can still be used on local databases.

Also switched Flyway Gradle tasks to get credentials from
new locations (in domain-registry-dev).

Updated README file on schema push process. Also reformatted
the entire file.
2019-11-15 11:44:21 -05:00
Lai Jiang
d55230933b
Upgrade to Gradle 6.0 (#364) 2019-11-13 13:17:29 -05:00
Shicong Huang
09aef04117
Add PersistenceXmlUtility and refactor related code (#357)
* Refactor GenerateSqlSchemaCommand

* Add and throw UncheckedClassNotFoundException
2019-11-12 15:08:54 -05:00
gbrodman
a392100852
Add by-repo-id method to RegistryLockDao (#344)
* Add by-repo-id method to RegistryLockDao

When we create new registry lock objects on user request, we will want
to make sure that there are no pending lock/unlock actions on this
domain. In order to do that, we will need to know what lock actions
there have been for this domain.

* Inline the query into a single statement

* whoops

* comments
2019-11-12 11:47:59 -05:00
Weimin Yu
b005e3aeb0
Fix a few lint errors (#361)
Replace deprecated bouncycastle class in SslInitializerTestUils.

Generic array as vargs: worked around it in ProbingAction and
removed unused method in CircularList.
2019-11-12 11:14:51 -05:00
Weimin Yu
bce09a3aa3
Deploy SQL schema from Cloud Build (#350)
Defined Docker image for schema deployment.

Included schema deploymer docker in the Cloud Build release process.

Defined Cloud Build config for schema deployment.

TESTED=Used cloud-build-local to test deployment flow
TESTED=Used docker to test schema deployer image in more ways
2019-11-12 11:06:16 -05:00
Weimin Yu
cb87658827
Remove unnecessary usage of CircularList (#358)
Implementation of CircularList is an overkill for its stated
purpose. Remove an unnecessary usage so that the remaining
use case may be refactored in isolation.
2019-11-11 18:04:46 -05:00
Michael Muller
455daae25c
Add converter for DateTime (#346)
* Add converter for DateTime

* Added to sql integration test suite

* Removed obsolete "auto update schema" property
2019-11-11 17:24:20 -05:00
gbrodman
dea7dfcf28
Refactor UI actions to be more reliable (#348)
* Refactor UI actions to be more reliable

- Created HtmlAction to handle the nitty-gritty of login and setting up
HTML pages
- Created a test to verify that all UI actions implement JSON GET, JSON
POST, or HTML classes
- Move CSS renaming into a utility class

* Move logging of request into HtmlAction

* Comment and wording in exception

* mention JsonGetAction in the comment

* JsonGetAction extends Runnable
2019-11-11 16:18:48 -05:00
Shicong Huang
3a47fa2fe9
Fix beam pipeline deployment issue (#360) 2019-11-11 15:57:33 -05:00
gbrodman
0b7a0ba99d
Update puppeteer (#359)
* Update puppeteer

They released 2.0.0 which fixes the security vuln in https-proxy-agent
2019-11-11 15:38:44 -05:00
gbrodman
ae89a8b76f
Run both the AppEngineRule and JpaTransactionManagerRule in tests (#356)
* Run both the AppEngineRule and JpaTransactionManagerRule in tests

As previously written, the AppEngineRule wraps the infinite loop of the
server handling requests. This changes the code so that we'll have a
chain that does both the AppEngine management and the JPA transaction
manager management so we can actually use SQL in tests and in the test
server.

* remove log line
2019-11-11 11:26:20 -05:00
Shicong Huang
f0345ddf89
Resolve test flakiness caused by connection leak (#355) 2019-11-08 15:40:27 -05:00
Weimin Yu
e9f20b3401
Delete obsolet gradle config in root project (#354)
* Delete obsolet gradle config in root project
2019-11-08 14:32:08 -05:00
Lai Jiang
64e7a593ef
Make Joda Money embeddable in entities (#340) 2019-11-07 17:03:00 -05:00
gbrodman
135ab66e55
Throw an EPP exception when using discount on premium domains (#351)
We should communicate to the users why this command failed, that they
are not allowed to use discounted allocation tokens on premium domains.
Currently it still fails, but we don't yet tell them why.
2019-11-07 15:30:23 -05:00
sarahcaseybot
69acb9f6de
Add cursors for tracking ICANN report uploads to Cursor.java (#349) 2019-11-07 10:49:21 -05:00
gbrodman
5359f4640a
Clean up Karma config file (#345)
Because we moved it into the source dirs, we don't need to reference
soyutils_usegoog.js separately
2019-11-06 10:33:58 -05:00
gbrodman
5c2856e3e2
Add RegistryLockGetActionTest to SQL testclasses (#347) 2019-11-05 15:10:13 -05:00
gbrodman
86e1fb85b6
Add a GET action and tests for registry lock retrieval (#326)
* Add a GET action and tests for registry lock retrieval

* Create isVerified method

* Allow lock access for admins even if they're not enabled on the registrar

* Simple CR responses

* Move locks retrieval to the GET action

* add newline at eof

* Switch to using ID
2019-11-05 13:19:32 -05:00
Michael Muller
301ab54fb4
Improve FilteringTest so it works in all cases (#339)
* Improve FilteringTest so it works in all cases

- Disable failOnMatchingTests so we don't fail if we filter out all of the
  tests for any given task.
- Add excludeTestCases flag so that we can turn this behavior off for test
  tasks that need to run TestCas/TestSuite classes.
- Add includeAllTests() function for test tasks where we don't explicitly
  include our required tests.

This makes all test tasks in core FilteringTest, with the exception of the
default test task which now does nothing (it just depends on the other test
tasks).

* Improve wording of excludeTestCases comment
2019-11-05 12:04:56 -05:00
Shicong Huang
2c6d71f04c
Prevent test from changing golden schema (#337)
When we add the extra test entity to the current JpaTransactionManagerRule by calling jpaRule.withEntityClass(TestEntity.class) and jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update"), the rule would override the golden database scheme with the schema derived from the current entity class(This is an expected behavior by enabling HBM2DDL_AUTO). This behavior prevents us from detecting breaking changes in ORM classes.

This PR fixed the issue by adding HibernateSchemaExporter to export the DDL script for given extra entity class, and make JpaTransactionManagerRule execute the DDL script to create the corresponding table for the extra entity when initializing the database. This PR also simplified the code when adding extra entity class for testing. For now, you don't need to(and shouldn't) call jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update").
2019-11-05 11:36:03 -05:00
Shicong Huang
ee4e6ace2a
Remove jdbcUrl config for nomulus tool (#330) 2019-11-05 10:33:53 -05:00
Lai Jiang
c4721121eb
Move soyutils_usegoog.js out of node_modules (#342)
* Move soyutils_usegoog.js out of node_modules

Everytime the npmInstall runs, it removes this file from node_modules.
Move it outside the folder to prevent this from happening.

* Move karma.conf.js and soyutils_usegooge.js

* Move karma.conf.js to be under core
2019-11-04 15:22:32 -05:00
Weimin Yu
37b9999cfc
Release SQL schema in Cloud Build (#341)
* Release SQL schema in Cloud Build

Tentatively release SQL schema at the same time as the server release.
Publish schema jar to gs://domain-registry-maven-repository/nomulus
and also upload it with server artifacts.

Also removed the Gradle 'version' variable which is not used.

Tested=On cloud-build with a simplified version of
cloudbuild-nomulus.yaml.
2019-11-04 10:22:05 -05:00