Commit graph

2590 commits

Author SHA1 Message Date
mcilwain
94a2681127 Send deletion poll messages when requested by superuser
Otherwise, registrars will never receive a notification through EPP that a
domain has been synchronously deleted by us.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=234172289
2019-02-18 17:34:36 -05:00
mcilwain
067756722d Change nomulus registrar --drive_id parameter to --drive_folder_id
This makes it consistent with the parameter of the same name on the tld commands.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=234148699
2019-02-18 17:32:57 -05:00
jianglai
b582614d7a Add the builder folder to the open source code base
It should have been part of []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=234036893
2019-02-18 17:31:19 -05:00
jianglai
23ee5b21ec Add a Dockerfile to build the builder
Our Gradle build now requires three programs to build: Java, npm and gcloud. There are no existing images that contain all of them. Even if there were, they probably come from some random Joe on the Internet and we cannot trust the image to be free of malwares. Therefore we need to build our own builder.

The builder images will be built by Cloud Build and upload to our container registry. We should periodically rebuild it to pull in the latest security updates both for the base Ubuntu image, and for the components that we install. I have not figured out a way to do that yet. For now we'll just trigger Cloud Build manually once in a while.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=234009343
2019-02-14 16:17:03 -05:00
gbrodman
063197fd69 Clean up the Gradle JS distribution a bit
There's no reason not to always create the source mapping but we shouldn't
distribute it in production.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233984970
2019-02-14 16:12:10 -05:00
jianglai
4241c7658f Add admin page link to create new registrar
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233956155
2019-02-14 16:08:57 -05:00
jianglai
9a6a57c871 Add gitignore rules for node modules
The npmInstall task installs gradle/node_modules/google-closure-library, which should not be tracked by git.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233826415
2019-02-14 16:05:37 -05:00
jianglai
f677373a54 Fix overrides of plugin repository (take 2 of []
The correct way to override the plugins repo is through the pluginManagement
section in the gradle settings file.  Also make use of the gradle.properties
file to initialize repositoryUrl and also publishUrl so we don't have to mess
around with finding and assigning them in the main gradle file.

The lock files are also updated.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233810854
2019-02-13 16:24:16 -05:00
mcilwain
4f9bce1686 Automated g4 rollback of changelist 233778270
*** Reason for rollback ***

Breaks the build.

*** Original change description ***

Fix overrides of plugin repository

The correct way to override the plugins repo is through the pluginManagement
section in the gradle settings file.  Also make use of the gradle.properties
file to initialize repositoryUrl and also publishUrl so we don't have to mess
around with finding and assigning them in the main gradle file.

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233801411
2019-02-13 16:19:18 -05:00
mmuller
287824564c Fix overrides of plugin repository
The correct way to override the plugins repo is through the pluginManagement
section in the gradle settings file.  Also make use of the gradle.properties
file to initialize repositoryUrl and also publishUrl so we don't have to mess
around with finding and assigning them in the main gradle file.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233778270
2019-02-13 16:17:38 -05:00
shicong
76028ba1b4 Partially externalize WebDriver tests
This change does a few things:

  1. Partially externalized WebDriver tests by using ChromeDriver
     as an implementation of WebDriver API in the external build.
  2. Refactored WebDriverRule.java to decouple the creation and
     using of WebDriver related stuff so we can have different
     implementations in internal and external builds.
  3. Refactored the usage of some internal libraries to have a
     central place to store all of them to make it easier to
     remove them in the external build.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233661757
2019-02-13 16:11:02 -05:00
guyben
4097dae3b2 Only inject @Parameter-created variables in the Action itself
Icann reports have 3 parameter-provided injections:

- yearMonth
- subdir
- reportType

We move all of them away from the "inner classes" and only @Inject them in the Actions themselves.

This has 2 benefits:
- it's much clearer what all the parameter inputs of the Actions are
- the "inner injected classes" don't assume anything about the Action that uses them - they will work just as well for JSON actions as for "regular" actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233625765
2019-02-13 16:07:45 -05:00
guyben
e6c46cab58 Set up the plugin build environment
The goal of this CL is to set up the build environment to allow plugins to work.

We have a trivial plugin that doesn't do anything (yet) - it just sets itself as the finalizer of all Reporting tasks.

Eventually, this plugin will upload all reports to GCS, and even create a "cover page" linking to each one of them.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233617499
2019-02-13 16:06:05 -05:00
jianglai
75c80b5e24 Create WAR files ready for deployment
This makes sure that the WAR files created by running "gradle stage" can be deployed by appcfg (tested the pubapi service on alpha). We need to copy all the static html files regardless of the service because the error.html handler is registered for sandbox and production environments across services. Without those files the gradle app engine plugin refuses to create the WAR files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233608424
2019-02-12 11:26:28 -05:00
jianglai
0b3fee4ef1 Update dependency lock files
Also upgrade gradle wrapper to the latest version (5.2.1)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233468712
2019-02-12 11:21:30 -05:00
mmuller
36605849ea Move the lockfiles back to where they should have been
It looks like I must have somehow duplicated the absolute path when I copied
these, this puts them in the right place (I think...)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233412326
2019-02-12 11:18:09 -05:00
jianglai
b19563f69d Use Maven repository on GCS for Cloud Build
This CL changes the Cloud Build flows to retrieve dependencies from our self-hosted GCS repository, to ensure that the release build are reproducible and hermetic (Note that it is still not truely reproducible as the dependency publishing process will override any existing artifacts in GCS with the current artifacts in Maven central. This is an issue that we should fix later).

There are a couple of changes involved to get this working:

1. Changed internal repo location to pull from the new repo.

2. Remove jcenter repo. It is only used to pull in the docker gradle plugin, which is not used. We instead build the deploy jar file with Gradle and build the docker image with a Dockerfile. The docker gradle plugin artifacts uploaded to GCS cannot be read because it is using some special classifier which seems to not be preserved when uploading. The java application plugin is also removed because it is only used by the docker gradle plugin.

3. Removed netty tcnative library classifier. It does not appear to be actually used (the jar downloaded from Maven central is an uber jar) and the classifier again interferes with downloading the artifacts from GCS.

4. Removed the cyclic dependency of the util project on itself. It was added because the nebula linter wanted it, which I think is an erroneous warning which should be reported upstream. The cyclic dependency was not a problem before (for yet unknown reasons), but it seems like when we force the dependency resolution (by calling project.generateDependencyPublications during configuration stage) it exacerbated the hidden issue and caused a cyclic task dependency in the util project, which is fatal. Now Nebula will complain again, but the warning is considered benign and will not cause the build to fail.

5. Added the nebula dependency lock files. We need these files when using the GCS maven repo because the we only upload artifacts after conflict resolution to GCS. If both v1 and v2 of the same library are requested in the dependency graph, only one will be uploaded. If we do not have the lock files in place, when building from GCS maven repo, Gradle will try to first find both v1 and v2 in the repo (which fails because v1 is not present in the repo), before proceeding to select v2 to use.

6. Refactored the code to upload Maven artifacts to GCS. We need to manually edit the POM file to reproduce the dependencies for each artifact so that they are all put in the classpath during compilation. Before, the POM files do not have any dependency information, which causes compilation to fail because transitive dependencies are not loaded (even though they are present in the GCS repo).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233408051
2019-02-11 11:24:12 -05:00
mcilwain
49ac4e3e69 Improve Datastore efficiency of duplicate contact messages
I should have caught this in the review, but [] is loading *ALL*
contacts individually from Datastore on every domain update. This will add a
large number of Datastore round trips and thus significantly reduce update
performance.

This CL changes the behavior to *ONLY* load contacts when there is a duplicate
(which is needed to determine the contact's display name to generate the error
message), and loads all of them in a single batch rather than individually.

This also makes some minor changes around domain getters returning empty sets
instead of null.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233128140
2019-02-11 11:22:34 -05:00
gbrodman
920d5d0190 Add source maps in to the JS output
This allows us to have the source mapping appear when debugging, just like we have when logging on to alpha, prod, or locally.

Tested on Crash with building from Gradle

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233092769
2019-02-11 11:19:24 -05:00
gbrodman
a995fc09cb Add the Closure Compiler/Library/Templates dependencies
This CL does a few things:
- Adds the template Soy-to-JS compilation (note: this requires the extra
  soyutils_usegoog.js file separately so that the compiled *.soy.js files work
- Adds the Closure Compiler to compile and check our JS
- Adds an NPM task to allow us to download dependencies
- Adds the Closure library as an NPM package

Note: this probably won't compile until we fix the test JS files

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233059414
2019-02-08 11:21:43 -05:00
shicong
71d65ed73a Add detailed log when DuplicateContactForRoleException is thrown
This change also added a test to verify that EPP request to modify
both contacts and registrant at same time can be handled as expected.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232935690
2019-02-08 11:20:21 -05:00
weiminyu
1c837dcad1 Declare inputs and outputs in code-generation tasks
This makes such tasks work with Gradle's incremental build.

In practice this change is not very useful:
- The Dagger annotation processor version we use is not fully
  compatible with Gradle incremental build, and currently causes
  unnecessary rebuild in the util package, and forces tests to always run.
  Will try the latest version later, which claims to support incremental
  build.
- AppEngine deploy task leaves behind non-writable tools.jar files.
  Reruns of any builds that include :services:*:explodeWar would fail,
  and the easiest work-around right now is to run Gradle clean task.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232929552
2019-02-08 11:18:59 -05:00
shicong
c2445c3797 Rename loadByForeignKeys to loadByForeignKeysCached
This is to remind the user that the function actually uses cache, and also
for naming consistency with EppResourceUtils.loadByForeignKeyCached().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232870846
2019-02-08 11:17:38 -05:00
gbrodman
93ff541006 Handle sunrise reservations properly during sunrise TLDs
We figure out the TLD state so that we properly check whether or not we can provision sunrise domains in that TLD. We also change the message slightly so that it's a bit more clear when we aren't in sunrise.

Note: it is deliberate that NAME_COLLISION reservations are provisionable in sunrise.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232742813
2019-02-06 18:06:21 -05:00
weiminyu
abe8a0ca0f Update dependency-related Gradle tasks
The publish and generateDependencyMetadata tasks need to run
on root project as well.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232742630
2019-02-06 18:04:47 -05:00
jianglai
7fe16689d8 Change redacted output for domain whois query
The redacted text for the email field displays a longer prompt to
contact the registrar, per the request filed at b/123573370.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232716133
2019-02-06 18:03:11 -05:00
mcilwain
6a6c7e3b09 Explain SERVER_HOLD part of NAME_COLLISION reservations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232713573
2019-02-06 18:01:37 -05:00
mmuller
a608e9f795 Check in initial dependency lockfiles.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232692353
2019-02-06 18:00:02 -05:00
mcilwain
29d3ad8052 Switch to constructor injection in a few actions
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232552406
2019-02-06 17:53:49 -05:00
guyben
927e8bbd73 Move LocalDate injection to the Actions themselves
We want to make it clear what query (or POST) inputs the user needs to / can give for each Action. That means moving all the @Injects of these parameters to the Actions themselves instead of injecting them in "hidden" indirect dependencies.

This has the extra benefit of allowing these indirect dependencies to work for JSON Actions as well, since the "regular" way we @Inject parameters can corrupt the POST JSON data.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232540758
2019-02-06 17:52:16 -05:00
gbrodman
89802329b3 Run more formatting fixes in preparation for the open-source change
Similar to [] these are issues found by Google Java Format. Most of the output is just using the standard [] formatter, then fixing any line-length issues.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232488284
2019-02-06 17:49:11 -05:00
gbrodman
058ef07e16 Fix some style issues that will help us out later
These files will have errors later when we run the Google Java Format plugin over their entirety (e.g. a situation where fixed indentation leads to a line that's longer than 100 characters). It's simpler to fix them now so we won't have to fix them later.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232353791
2019-02-06 17:47:37 -05:00
weiminyu
a325f0a50b Use Gradle native dependency-locking mechanism
Remove Nebula plugin and enable Gradle locking.

Update publishing task to include build dependencies in resolved artifacts
list.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232349134
2019-02-06 17:46:04 -05:00
mcilwain
339202e879 Delete the mapreduce poll action
We never used it for anything.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232024489
2019-02-01 16:24:37 -05:00
mmuller
7596a0b773 Delete unused testdata files
These files are unused by the tests in this directory, have no obvious purpose
and have no README explaining their existence.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232014732
2019-02-01 16:23:05 -05:00
mmuller
effbf836fd Add BaseDnsWriterTest to opensource tree
This should have gone in with the implementation of BaseDnsWriter.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232009971
2019-02-01 16:21:32 -05:00
mcilwain
f01ded3fa7 Make logged response in NordnUploadAction intelligible
Right now it's logging the raw bytes, which look like:

response data: [65, 117, 116, 104, 111, 114, 105, 122, 97, 116, 105, 111, 110, 32, 114, 101, 113, 117, 105, 114, 101, 100]

We'd rather convert it to ASCII characters (what the NORDN service uses) before
logging it, so that it'd instead look like:

response data: Authorization required

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231998658
2019-02-01 16:20:01 -05:00
guyben
c5ad30f49d Prevent spellchecking from textarea fields
The spellchecking causes test flakiness.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231984097
2019-02-01 16:18:24 -05:00
weiminyu
8085468280 Lock transitive dependencies in Gradle project
By default, only direct dependencies are locked.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231821158
2019-01-31 14:34:30 -05:00
weiminyu
8115909099 Add deploy notes to Gradle README
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231612510
2019-01-31 14:03:45 -05:00
mmuller
96f0c519f6 Move update_dependency.sh to g3/d/r/tools
We don't need this script to be publicly visible.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231606840
2019-01-31 14:02:15 -05:00
mcilwain
3812c2ceda Use -dot- subdomain notation in MapReduce console links
appspot.com is not provisioned with a multi-level wildcard SSL certificate, so
URLs of the form https://service.projectid.appspot.com/path need to be rewritten
as https://service-dot-projectid.appspot.com/path (and same for version names).

This is a follow-up to []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231418795
2019-01-31 13:59:05 -05:00
gbrodman
6e0b8f2cfd Pass the date field in to the Spec11 Soy template
The daily template is the only one that needs it but we can always pass it in without issue.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231295089
2019-01-31 13:57:30 -05:00
mcilwain
2edb072e68 Delete the deprecated RegistrarBillingEntry Datastore index
This should have been deleted in [] when the underlying entity was
deleted, but it was missed. It's been a no-op since then. This is just cleanup.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231293033
2019-01-31 13:55:52 -05:00
mcilwain
88aa13d1a4 Log a warning when MarksDB password isn't configured
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231287550
2019-01-31 13:51:10 -05:00
gbrodman
44e822ed86 Add Checkstyle and Error Prone to the Gradle build
We make some modifications to the internal Google checkstyle file because Google's linter uses a modified build of Checkstyle that introduces some new classes and allows for more specific checks than the open-source Checkstyle (e.g. only enforcing UPPER_SNAKE_CASE on deeply immutable fields). There exists a public Checkstyle file that purports to be the Google java format file (https://github.com/checkstyle/checkstyle/blob/master/src/main/resources/google_checks.xml) but it doesn't quite match up with what the internal linter says in certain situations (e.g. what operators must/can appear on new lines).

The suppressions are basically "don't run on generated code + don't care about Javadoc in test code"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231283700
2019-01-31 13:49:33 -05:00
Michael Muller
edc29d1a5a Adding png file to test image diffs in reviewable. 2019-01-31 12:40:26 -05:00
mcilwain
5dedc1e889 Delete everything related to RDE import
This code was never finished or fully working anyway.  It would require
substantial reworking for the Registry 3.0 migration because it's closely tied
to the Datastore model and App Engine MapReduce framework, both of which will be
going away.  We can bring back some of these deleted test files as necessary
if/when we rewrite RDE import for the new schema.

On the plus side, in a relational database, RDE import will be much simpler.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231265578
2019-01-28 16:16:36 -05:00
mcilwain
f0c677b18b Rename DNL and SMDRL "login" to "loginAndPassword"
They are passed around in the format username:password, whereas just saying
"login" implies it's just a username and not necessarily also a secret
password. Putting password in the variable name makes it obvious what this is
and reduces the likelihood of anyone ever logging it or otherwise using it
inappropriately.

Note that this does not require data migrations as the actual key used to store
the data in KMS remains unchanged.

This is a follow-up to []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231253964
2019-01-28 16:15:04 -05:00
mcilwain
0130f91830 Add logging to DNL fetch action
This will help us to debug the current MarksDB issue.  This also throws an explicit error earlier when attempting to connect to MarksDB without login credentials being specified, which we know will fail.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231236317
2019-01-28 16:13:27 -05:00