Commit graph

96 commits

Author SHA1 Message Date
guyben
9aa7b69921 Add web console for creating registrars
This console is only to be used by Admins (either GAE admins for this project, or Support accounts). It is for "internal" use only, not for use by the registrars themselves.

To prevent abuse, the registrar is created in a non-functional PENDING state and can only be made functional from the nomulus shell tool.

While in "PENDING" state, the registrar can be updated from the registrar-console by admins.

Also - moving all the web consoles to the same directory (moving the otesetup/* files into registrar/)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229681011
2019-01-17 19:19:09 -05:00
weiminyu
758dcb5c39 Fix Gradle lint's dependency warnings
This change fixes the following warnings when applied to cloned
github nomulus:

warning   undeclared-dependency              one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly

warning   undeclared-dependency              one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly

warning   unused-dependency                  one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly

warning   unused-dependency                  one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly

✖ 4 problems (0 errors, 4 warnings)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229629444
2019-01-17 19:17:32 -05:00
weiminyu
bc798f33e9 Automated g4 rollback of changelist 229414759.
*** Reason for rollback ***

Found more tests failing.

*** Original change description ***

Reenable Test Executor sharing in Gradle build

Combining all tests in one suite and drop the forkEvery=1 directive.
Issue was fixed by [] and []

TESTED=Run locally with maxParallelForks =1 and 5, and tested on travis
       with maxParallelForks=5

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229430425
2019-01-17 19:09:29 -05:00
weiminyu
035b9149e0 Reenable Test Executor sharing in Gradle build
Combining all tests in one suite and drop the forkEvery=1 directive.
Issue was fixed by [] and []

TESTED=Run locally with maxParallelForks =1 and 5, and tested on travis
       with maxParallelForks=5

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229414759
2019-01-17 19:04:08 -05:00
jianglai
98f11decb9 Update Gradle to 5.1.1
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229240423
2019-01-14 16:32:49 -05:00
gbrodman
f017798162 Add diff logic and send daily Spec11 emails with new threats
For each registrar, the daily email will only include threats that did not appear
in the prior run's email.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228889972
2019-01-14 16:20:04 -05:00
mcilwain
a6476862fd Use Guava instead of Apache Commons for file ops in Ghostryde
This backs out most of [] fixes the external build (which wasn't
finding Apache Commons correctly), and makes miscellaneous tweaks and fixes,
including better handling representing the default case of decrypting to stdout.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228877090
2019-01-11 11:00:33 -05:00
mcilwain
580302898d Delete end-date sunrise, landrush, and sunrush phases
This also deletes the associated commands and domain application specific
entities.

We haven't used any of these TLD phases since early 2015 and have no
intent to do so in the future, so it makes sense to delete them now so we
don't have to carry them through the Registry 3.0 migration.

Note that, while there are data model changes, there should be no required
data migrations. The fields and entities being removed will simply remain
as orphans. I confirmed that the removed types (such as the SUNRUSH_ADD
GracePeriodType) are no longer used in production data, and left types
that are still used, e.g. BillingEvent.Flag.LANDRUSH or
HistoryEntry.Type.ALLOCATE.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228752843
2019-01-10 16:23:35 -05:00
mmuller
8bde7285bf Fix travis build with test isolation
Since we seem to be experiencing weird interference between tests on Travis,
run every test in its own JVM until we find a better solution.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228738848
2019-01-10 16:23:35 -05:00
shicong
4866955c76 Unify dependency declarations into one place
This change unified declarations of third party dependencies into
one dependencies.gradle file by reference to this blog
https://medium.com/freelancer-engineering/managing-dependencies-in-multi-project-builds-with-gradle-7626d9c6448d

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228713862
2019-01-10 16:23:35 -05:00
jianglai
ce29e73879 Upgrade to Gradle 5.1
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228520681
2019-01-10 16:23:35 -05:00
Michael Muller
807fe73db4 Fork for every test to verify that this fixes the travis build. 2019-01-09 11:13:48 -05:00
mmuller
9e0b675275 Don't use Files.copy() when decrypting to stdout
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout.  Instead copy directly from the decoder
to standard output.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
2019-01-08 16:51:11 -05:00
shicong
04a495bc99 Fix Gradle build
There were two issues in the Gradle build.

 1. Missing soyToJava conversion for newly added
    google.registry.ui.soy.otesetup.

 2. testSuccess_setAllowedTldsUncached_newTldNotInCache in
    RegistrarTest modified the cache duration to 600 seconds
    which broke other tests as those tests write to the storage
    in initiation stage and expect the data to be available in
    the cache immediately, this requires the duration to be
    0.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228377715
2019-01-08 16:49:39 -05:00
jianglai
c3de49130a Fix Gradle build
Added SOY templates for OTE console need to be complied.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228255528
2019-01-08 11:06:51 -05:00
jianglai
a09067c51d Make Cloud Build produce artifact ready for Spinnaker to consume
The artifacts for each service will be packaged inside a tar file that, when untared, is ready for Spinnaker to deploy to GAE.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226516223
2018-12-21 15:55:08 -05:00
mmuller
3d3c0eb9ae Add a set of newly broken tests to the "outcasts" group
Recent changes cause a bunch of new tests to have runtime conflicts.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226499557
2018-12-21 15:55:08 -05:00
shicong
40b05ffb3c Add a script to generate dependency metadata
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225872347
2018-12-20 07:46:33 -05:00
mmuller
214fb49091 Add "showAllOutput" project property
Added a property to show all test output, useful for debugging.

Usage: ./gradlew -PshowAllOutput=true build
Tested:
  Verified that the property does the right thing for true and false values
  and verified that the build doesn't show test output if it's omitted.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225854375
2018-12-20 07:46:33 -05:00
mmuller
2ec8246097 Add a dependency on the local tools.jar for doc tool generation.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225565675
2018-12-20 07:46:33 -05:00
jianglai
f96da3cb29 Add a cloud build config to upload WAR files to GCS
The private repo structure is re-done to mirror that of the public repo to facilitate easy merging.

Also removed steps to tag the private repos. This will likely cause a race condition if both the nomulus and the proxy cloud build are triggered by the same tag. They will both try to tag the private repo with the same tag.

The tagging of the private repo should happen simultaneously with the public repo tagging, and in an out-of-band process as far as the build process is concerned. The build process should not have side effect on its source.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225544063
2018-12-14 11:29:46 -05:00
shicong
296acf80bb Add build logic to upload dependency JARs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225539482
2018-12-14 11:28:10 -05:00
jianglai
377736e987 Add an option to perform login on remote machines
If the user runs "nomulus -e [ENV] login --remote", an URL will be provided, the user then can visit the URL on any machine (not necessary where the command is run) and copy&paste back the authorization code to complete authorization.

This makes it easy to login on machines where local browsers are not easily accessible.

Also upgraded nebula lint version to 10.3.5.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225198700
2018-12-12 13:22:34 -05:00
jianglai
c7fc964d9c Add a Gradle task to build the nomulus tool
It'd be nice if we can separate out the tool to its own package and reduce the transitive dependencies that it pulls in. However since the entire core project is a dependency of the tool, it doesn't make any difference as we'd be pulling in core and all its transitive dependencies as well.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224821931
2018-12-12 13:22:34 -05:00
jianglai
a85544b3f6 Use gson to make JSON string in proxy log formatter
This is simpler than using fasterxml.jackson.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224583713
2018-12-12 13:22:34 -05:00
jianglai
9735978b87 Consolidate app engine deployment gradle tasks
Removed duplicate build files for each service, and also added two tasks that runs deployment/staging for all services.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224359198
2018-12-06 13:34:16 -05:00
jianglai
a612e9bf66 Use local credential to deploy beam pipelines
We are moving away from using Application Default Credentials generated by "gcloud auth application-default login" in our code base and consolidate on using self-managed credentials provided from AuthModule.

One of the remaining dependencies on the ADCs is from beam pipeline deployment commands, which by default use the ADCs to talk to GCS and upload the jar files and templates. In this CL, we explicitly provide the locally created credential to the Options used in deployments.

Also moved all credential qualifiers to CredentialModule, and removed @AppEngineAdminApiCredential, which is no longer used.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224199812
2018-12-05 16:07:54 -05:00
jianglai
6352b8a01a Use self-managed credential in remote api installer
RemoteApiOption has a package-private method that takes a Stream representing the content of a JSON and use a GoogleCredential created from it as its credential. This CL uses reflection to change the access modifier of that method in order to supply a credential stream that is self-managed. This is obviously not ideal and prone to breakage in case the getGoogleCredentialStream method is changed. Unfortunately upstream is not willing to make it public citing the reason that GoogleCredential.fromStream() (which getGoogleCredentialStream uses) is a @Beta annotated function (see https://groups.google.com[]forum/#!searchin/domain-registry-eng/remoteapioptions%7Csort:date/domain-registry-eng/Flsah6skszQ/CySZv2XEBwAJ). However this function is introduced 5 years ago as a public function (b857184bfa). I think at this point it is safe to assume that it is part of the widely used APIs and will not change without sufficient notice.

Note here that RemoteApiOptions creates its own copy of GoogleCredential to be used to call App Engine APIs locally, whereas communications to Nomulus endpoints use the Credential provided in AuthModule. Even though both credentials are created from the same client id, client secret and refresh token (the three elements needed to construct a GoogleCredential this way, see https://github.com/googleapis/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/GoogleCredential.java#L842), their refreshes cycles are independent of each other. I verified that refreshing one of the credential does not invalidate the access token of the other credential, as long as it is not expired yet.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224156131
2018-12-05 16:02:28 -05:00
jianglai
7ec9513f3f Check gradle wrapper in to VCS
The version of gradle used in compilation is now controlled by the gradle
wrapper and is centrally managed. This means that everyone will be using the
same version of gradle for build, including IC systems like Travis and Cloud
Build.

The Gradle version for this Gradle wrapper is v5.0.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223856122
2018-12-03 19:26:38 -05:00
mmuller
ec11eae699 Add App Engine Deploy to Gradle build
Add a first pass add Nomulus GAE deployment to our gradle build scripts.  This
attempt could stand some improvement.  In particular:

- It currently only deploys to crash, per discussion we will make the
  environment a parameter.
- This uses a different import mechanism from the other plugins (via a
  "buildscript" section) and it looks like the lint plugin is very similar.
  I'll experiment with getting it to work that way.
- There is a lot of redundancy in the deployment scripts for each module, and
  given that we have the full power of a general programming language it
  should be possible for us to reuse the common parts.

But that said, this is a pretty good first step and I wanted to put it out
there.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223366370
2018-12-03 19:18:44 -05:00
shicong
d20b83c820 Add option --non_live_versions to set_num_instances command
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
2018-12-03 18:58:05 -05:00
jianglai
f46f817f9e Create proxy Docker image with Gradle
This CL adds two ways to build an docker image with gradle:

1) Adds a :proxy:deployJar task that builds an uber jar that contains all runtime dependencies. The jar can then be add to a docker image by calling docker with the added Dockerfile. The base image for this image can be both distroless java or openjdk:alpine.

2) Uses the Gradle distribution plugin to build a distribution tar file that contains all dependencies (as separate jar files) and a run script that sets up the classpath before calling the main class. Then the docker application plugin can build a docker image (with the dockerBuildImage task) using the application tar file. This only works with openjdk:alpline base image as the distroless java image does not contain a shell and therefore the script created by the distribution plugin cannot be launched.

We may later decide to use one of the method and remove the other.

Also adds an outcast test pattern that caused the tests to be flaky.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222145192
2018-11-19 18:19:27 -05:00
jianglai
118367dc6b Move proxy to its separate project
It can be run directly by "gradle :proxy:run"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222102849
2018-11-19 18:17:47 -05:00
jianglai
997b49f7ca Move util to its separate project
It will make it easier later to have the proxy project depend on it, rather
than on the entire core project

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222081994
2018-11-19 18:16:03 -05:00
jianglai
f7e95e1ff1 Refactor gradle build script
1. Updated nebula lint version to 10.3.1
2. Do not apply plugins in the root projects that are not needed.
3. Only do linting when a build is successful, so that build failure message are not flushed by linter warnings.
4. Added explicitly stated java source and target version.
5. Moved source sets set up to the root project so that it applies to all subprojects. Currently there is only "core", but we will add at least "proxy" and "util" later, both of which will use mostly the same source sets (but with additional inclusion rules to only build classes in a specific sub folder in the source tree). By putting the set up closure inside the subprojects block in the root script, it can be reused.
6. Rename maybe_runtime configuration to maybeRuntime, which is consistent with other camelCase configuration names like testCompile.
7. Added a runtime dependency of the JAXB API which is removed from Java SE as of version 11.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222081092
2018-11-19 18:14:09 -05:00
jianglai
02a23c6ca6 Remove unnecessary source sets
These source sets are not configured correctly (no classpaths for example) and
they subsequently cannot compile. The special tests are put into separate tasks that
run from the same source set (test).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222080273
2018-11-19 18:12:12 -05:00
weiminyu
dee559baee Define multiple test suites under Gradle for better test performance
Tests are divided into three test suites without intra-suite conflicts.
This allows us to unset forkEvery=1 (4x improvement on my desktop) and
increase execution parallelism (additional 2x improvement on my desktop).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221656972
2018-11-16 16:55:55 -05:00
weiminyu
783c010ab4 Add a IDE helper plugin for annotation processors
Without the net.ltgt.apt plugin, Dagger does not retain generated source code,
and AutoValue mixes generated source code with classes. The new plugin fixes this
problem, making it easier to work with IDEs.

Based on limit experience, Intellij Idea can properly import this project. We
may consider adding IDE-specific plugins in the future.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220802052
2018-11-12 14:51:40 -05:00
weiminyu
6a870e5820 Check for unused dependencies
Enabled unused-dependency check using nebula-lint.

Dependencies that are not used by compile or testing are
labeled with 'maybe_runtime". We leave these dependencies
in the script for easy reference. Before launching Gradle-based
release process we must determine which of these should
be removed and which should be relabeled as runtime.

Label assignment:
- All dependencies recommended for removal from 'compile'
  are changed to maybe_runtime
- All dependencies recommended for move from 'compile' to
  testCompile are split into two lines, one with testCompile,
  the other maybe_runtime

Incidentally, Gradle 4.10.2 needs a groovy upgrade before
it can work with Oracle JDK 11.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219803797
2018-11-02 14:49:18 -04:00
weiminyu
00d0284d5b Add nebula-lint plugin to Gradle scripts
Updated the project config so that the plugin can work.

Reverted to the deprecated compile/testCompile labels
so that the plugin can work. This plugin provides valuable
checks that we do not want to give up.

Added undeclared-dependency check. Still need to add
unused-dependency and one-version check.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219406600
2018-10-31 19:15:03 -04:00
jianglai
c375b0d5f4 Add javax.activation as a dependency of jaxb
In Java 9+, this module is no longer loaded by default. JAXB needs it to compile.

Also changed the root project name to "nomulus".

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219000053
2018-10-29 15:41:06 -04:00
weiminyu
f6d9b46622 Reorganize Gradle dependencies
Move configs and dependencies from root script to subprojects.
There is not too much commonality between subprojects. Besides,
the gradle lint plugin (not added to this project yet) does not
handle inherited dependencies properly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218698818
2018-10-25 14:50:26 -04:00
jianglai
f0195f3ed6 Explicitly state which Gradle version is tested.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218687425
2018-10-25 14:45:30 -04:00
weiminyu
6f155ed3d2 Properly label compile-only dependencies in Gradle script
Dependencies labeled as 'neverlink' in bazel should have two
labels in Gradle: compileOnly and testCompileOnly/testImplementation.
This excludes the dependency from deployment archive.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218394354
2018-10-25 14:42:17 -04:00
weiminyu
b81525cea4 Set up gradle for Nomulus on github
Also updates apache-sshd dependencies.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217603216
2018-10-22 18:51:39 -04:00
weiminyu
fed4ec496f Add Gradle scripts to Nomulus
Experimental Gradle scripts that build and run all Java tests.

Please see README.md for status and limitations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217203690
2018-10-17 11:53:01 -04:00