Commit graph

131 commits

Author SHA1 Message Date
guyben
a03d10ce22 Add ability to save report to local folder
Updated the plugin to receive the "protocol"-like tag in the destination, so that you can choose whether to upload to GCS or just save it locally.

Possibly we might expand this in the future, but for now the goal was to allow saving our "internal" builds locally until we find a secure way to store AND BROWSE them remotely.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=238055136
2019-03-20 14:25:28 -04:00
shicong
bc3bdc7347 Upgrade the version of google/errorprone plug-in
This CL upgraded google/errorprone plug-in to 2.3.3 and resolved
some warnings detected from the plug-in.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=238047862
2019-03-20 14:25:28 -04:00
gbrodman
50e0a9b532 Refactor common email sending utility
The main thrust of this is to create a common POJO that contains email content in a simple way, then have one class that converts that to an email and sends it. Any class that uses email should only have to deal with creating that POJO.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237883643
2019-03-20 14:25:28 -04:00
shicong
9823ee7fcf Enable Webdriver test in Kokoro build
We changed to use Docker to provision Chrome browser and
ChromeDriverService, and the used Kokoro VM comes with
the docker CLI so we can enable the webdriver tests again
in kokoro build.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237874165
2019-03-20 14:25:28 -04:00
shicong
a7d099d2a3 Enable screenshot comparison in external build
This change actually enabled the screenshot comparison in the
visual regression tests. We used Docker to provision Chrome
and ChromeDriver to eliminate the discrepancy of environment
between local development and Travis CI.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237811918
2019-03-20 14:25:28 -04:00
shicong
984aed94a2 Update many dependencies to latest version
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237514773
2019-03-08 18:39:42 -05:00
weiminyu
eb8d80f7ad Ensure gradle build has only one servlet-api
Use javax.servlet:servlet-api:2.5 and exclude all other implementations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237505707
2019-03-08 18:38:09 -05:00
jianglai
b573b10ae1 Remove backported MockitoJUnitRule
Mockito in third_party is updated to 1.10. We do not need to backport this rule anymore.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237496086
2019-03-08 18:35:00 -05:00
guyben
193bd49406 Zip report folders that don't have a clear entry point
This is mostly for build reports that use xml or other non-browsable
format. Most notable - the JUnit xml test results.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=236118885
2019-03-05 14:19:12 -05:00
shicong
26490e607e Copy dependency BOMs to our GoB vomit repo
We currently deploy metadata files for vulnerability scanning
to a GoB repo. The Vomit team has requested that we provide
the .pom and .jar files for each dependency in the GoB repo
for their new manifest approach of scanning.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=235029408
2019-02-25 11:13:08 -05:00
guyben
6d9f718466 Upload task results to GCS
This uploads all the report files to GCS, and also creates a cover page that
summarises all the task states.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=234601015
2019-02-25 11:13:08 -05:00
Michael Muller
fc4268b470 Revert "Merge branch 'master' into master"
This reverts commit 9dfbfc5e32, reversing
changes made to 94a2681127.
2019-02-19 15:44:11 -05:00
Michael Muller
6dbb74c1d0 Merge branch 'master' into master 2019-02-19 15:28:01 -05:00
gbrodman
063197fd69 Clean up the Gradle JS distribution a bit
There's no reason not to always create the source mapping but we shouldn't
distribute it in production.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233984970
2019-02-14 16:12:10 -05:00
jianglai
f677373a54 Fix overrides of plugin repository (take 2 of []
The correct way to override the plugins repo is through the pluginManagement
section in the gradle settings file.  Also make use of the gradle.properties
file to initialize repositoryUrl and also publishUrl so we don't have to mess
around with finding and assigning them in the main gradle file.

The lock files are also updated.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233810854
2019-02-13 16:24:16 -05:00
mcilwain
4f9bce1686 Automated g4 rollback of changelist 233778270
*** Reason for rollback ***

Breaks the build.

*** Original change description ***

Fix overrides of plugin repository

The correct way to override the plugins repo is through the pluginManagement
section in the gradle settings file.  Also make use of the gradle.properties
file to initialize repositoryUrl and also publishUrl so we don't have to mess
around with finding and assigning them in the main gradle file.

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233801411
2019-02-13 16:19:18 -05:00
mmuller
287824564c Fix overrides of plugin repository
The correct way to override the plugins repo is through the pluginManagement
section in the gradle settings file.  Also make use of the gradle.properties
file to initialize repositoryUrl and also publishUrl so we don't have to mess
around with finding and assigning them in the main gradle file.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233778270
2019-02-13 16:17:38 -05:00
shicong
76028ba1b4 Partially externalize WebDriver tests
This change does a few things:

  1. Partially externalized WebDriver tests by using ChromeDriver
     as an implementation of WebDriver API in the external build.
  2. Refactored WebDriverRule.java to decouple the creation and
     using of WebDriver related stuff so we can have different
     implementations in internal and external builds.
  3. Refactored the usage of some internal libraries to have a
     central place to store all of them to make it easier to
     remove them in the external build.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233661757
2019-02-13 16:11:02 -05:00
guyben
e6c46cab58 Set up the plugin build environment
The goal of this CL is to set up the build environment to allow plugins to work.

We have a trivial plugin that doesn't do anything (yet) - it just sets itself as the finalizer of all Reporting tasks.

Eventually, this plugin will upload all reports to GCS, and even create a "cover page" linking to each one of them.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233617499
2019-02-13 16:06:05 -05:00
jianglai
75c80b5e24 Create WAR files ready for deployment
This makes sure that the WAR files created by running "gradle stage" can be deployed by appcfg (tested the pubapi service on alpha). We need to copy all the static html files regardless of the service because the error.html handler is registered for sandbox and production environments across services. Without those files the gradle app engine plugin refuses to create the WAR files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233608424
2019-02-12 11:26:28 -05:00
jianglai
0b3fee4ef1 Update dependency lock files
Also upgrade gradle wrapper to the latest version (5.2.1)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233468712
2019-02-12 11:21:30 -05:00
mmuller
36605849ea Move the lockfiles back to where they should have been
It looks like I must have somehow duplicated the absolute path when I copied
these, this puts them in the right place (I think...)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233412326
2019-02-12 11:18:09 -05:00
jianglai
b19563f69d Use Maven repository on GCS for Cloud Build
This CL changes the Cloud Build flows to retrieve dependencies from our self-hosted GCS repository, to ensure that the release build are reproducible and hermetic (Note that it is still not truely reproducible as the dependency publishing process will override any existing artifacts in GCS with the current artifacts in Maven central. This is an issue that we should fix later).

There are a couple of changes involved to get this working:

1. Changed internal repo location to pull from the new repo.

2. Remove jcenter repo. It is only used to pull in the docker gradle plugin, which is not used. We instead build the deploy jar file with Gradle and build the docker image with a Dockerfile. The docker gradle plugin artifacts uploaded to GCS cannot be read because it is using some special classifier which seems to not be preserved when uploading. The java application plugin is also removed because it is only used by the docker gradle plugin.

3. Removed netty tcnative library classifier. It does not appear to be actually used (the jar downloaded from Maven central is an uber jar) and the classifier again interferes with downloading the artifacts from GCS.

4. Removed the cyclic dependency of the util project on itself. It was added because the nebula linter wanted it, which I think is an erroneous warning which should be reported upstream. The cyclic dependency was not a problem before (for yet unknown reasons), but it seems like when we force the dependency resolution (by calling project.generateDependencyPublications during configuration stage) it exacerbated the hidden issue and caused a cyclic task dependency in the util project, which is fatal. Now Nebula will complain again, but the warning is considered benign and will not cause the build to fail.

5. Added the nebula dependency lock files. We need these files when using the GCS maven repo because the we only upload artifacts after conflict resolution to GCS. If both v1 and v2 of the same library are requested in the dependency graph, only one will be uploaded. If we do not have the lock files in place, when building from GCS maven repo, Gradle will try to first find both v1 and v2 in the repo (which fails because v1 is not present in the repo), before proceeding to select v2 to use.

6. Refactored the code to upload Maven artifacts to GCS. We need to manually edit the POM file to reproduce the dependencies for each artifact so that they are all put in the classpath during compilation. Before, the POM files do not have any dependency information, which causes compilation to fail because transitive dependencies are not loaded (even though they are present in the GCS repo).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233408051
2019-02-11 11:24:12 -05:00
gbrodman
920d5d0190 Add source maps in to the JS output
This allows us to have the source mapping appear when debugging, just like we have when logging on to alpha, prod, or locally.

Tested on Crash with building from Gradle

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233092769
2019-02-11 11:19:24 -05:00
gbrodman
a995fc09cb Add the Closure Compiler/Library/Templates dependencies
This CL does a few things:
- Adds the template Soy-to-JS compilation (note: this requires the extra
  soyutils_usegoog.js file separately so that the compiled *.soy.js files work
- Adds the Closure Compiler to compile and check our JS
- Adds an NPM task to allow us to download dependencies
- Adds the Closure library as an NPM package

Note: this probably won't compile until we fix the test JS files

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233059414
2019-02-08 11:21:43 -05:00
weiminyu
1c837dcad1 Declare inputs and outputs in code-generation tasks
This makes such tasks work with Gradle's incremental build.

In practice this change is not very useful:
- The Dagger annotation processor version we use is not fully
  compatible with Gradle incremental build, and currently causes
  unnecessary rebuild in the util package, and forces tests to always run.
  Will try the latest version later, which claims to support incremental
  build.
- AppEngine deploy task leaves behind non-writable tools.jar files.
  Reruns of any builds that include :services:*:explodeWar would fail,
  and the easiest work-around right now is to run Gradle clean task.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232929552
2019-02-08 11:18:59 -05:00
weiminyu
abe8a0ca0f Update dependency-related Gradle tasks
The publish and generateDependencyMetadata tasks need to run
on root project as well.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232742630
2019-02-06 18:04:47 -05:00
mmuller
a608e9f795 Check in initial dependency lockfiles.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232692353
2019-02-06 18:00:02 -05:00
weiminyu
a325f0a50b Use Gradle native dependency-locking mechanism
Remove Nebula plugin and enable Gradle locking.

Update publishing task to include build dependencies in resolved artifacts
list.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232349134
2019-02-06 17:46:04 -05:00
weiminyu
8085468280 Lock transitive dependencies in Gradle project
By default, only direct dependencies are locked.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231821158
2019-01-31 14:34:30 -05:00
weiminyu
8115909099 Add deploy notes to Gradle README
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231612510
2019-01-31 14:03:45 -05:00
mmuller
96f0c519f6 Move update_dependency.sh to g3/d/r/tools
We don't need this script to be publicly visible.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231606840
2019-01-31 14:02:15 -05:00
gbrodman
44e822ed86 Add Checkstyle and Error Prone to the Gradle build
We make some modifications to the internal Google checkstyle file because Google's linter uses a modified build of Checkstyle that introduces some new classes and allows for more specific checks than the open-source Checkstyle (e.g. only enforcing UPPER_SNAKE_CASE on deeply immutable fields). There exists a public Checkstyle file that purports to be the Google java format file (https://github.com/checkstyle/checkstyle/blob/master/src/main/resources/google_checks.xml) but it doesn't quite match up with what the internal linter says in certain situations (e.g. what operators must/can appear on new lines).

The suppressions are basically "don't run on generated code + don't care about Javadoc in test code"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231283700
2019-01-31 13:49:33 -05:00
shicong
9cd37189c2 Combine dependency update scripts
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230958523
2019-01-28 16:03:29 -05:00
weiminyu
d3e25cd5b1 Enable Test Executor sharing again in Gradle build
Identified a few more tests that are impacted by test cleanup
issues outside of Nomulus code base, more likely thread pool
cleanup behaviors in AppEngine testing fixture.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229934266
2019-01-18 15:32:33 -05:00
guyben
9aa7b69921 Add web console for creating registrars
This console is only to be used by Admins (either GAE admins for this project, or Support accounts). It is for "internal" use only, not for use by the registrars themselves.

To prevent abuse, the registrar is created in a non-functional PENDING state and can only be made functional from the nomulus shell tool.

While in "PENDING" state, the registrar can be updated from the registrar-console by admins.

Also - moving all the web consoles to the same directory (moving the otesetup/* files into registrar/)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229681011
2019-01-17 19:19:09 -05:00
weiminyu
758dcb5c39 Fix Gradle lint's dependency warnings
This change fixes the following warnings when applied to cloned
github nomulus:

warning   undeclared-dependency              one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly

warning   undeclared-dependency              one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly

warning   unused-dependency                  one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly

warning   unused-dependency                  one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly

✖ 4 problems (0 errors, 4 warnings)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229629444
2019-01-17 19:17:32 -05:00
weiminyu
bc798f33e9 Automated g4 rollback of changelist 229414759.
*** Reason for rollback ***

Found more tests failing.

*** Original change description ***

Reenable Test Executor sharing in Gradle build

Combining all tests in one suite and drop the forkEvery=1 directive.
Issue was fixed by [] and []

TESTED=Run locally with maxParallelForks =1 and 5, and tested on travis
       with maxParallelForks=5

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229430425
2019-01-17 19:09:29 -05:00
weiminyu
035b9149e0 Reenable Test Executor sharing in Gradle build
Combining all tests in one suite and drop the forkEvery=1 directive.
Issue was fixed by [] and []

TESTED=Run locally with maxParallelForks =1 and 5, and tested on travis
       with maxParallelForks=5

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229414759
2019-01-17 19:04:08 -05:00
jianglai
98f11decb9 Update Gradle to 5.1.1
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229240423
2019-01-14 16:32:49 -05:00
gbrodman
f017798162 Add diff logic and send daily Spec11 emails with new threats
For each registrar, the daily email will only include threats that did not appear
in the prior run's email.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228889972
2019-01-14 16:20:04 -05:00
mcilwain
a6476862fd Use Guava instead of Apache Commons for file ops in Ghostryde
This backs out most of [] fixes the external build (which wasn't
finding Apache Commons correctly), and makes miscellaneous tweaks and fixes,
including better handling representing the default case of decrypting to stdout.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228877090
2019-01-11 11:00:33 -05:00
mcilwain
580302898d Delete end-date sunrise, landrush, and sunrush phases
This also deletes the associated commands and domain application specific
entities.

We haven't used any of these TLD phases since early 2015 and have no
intent to do so in the future, so it makes sense to delete them now so we
don't have to carry them through the Registry 3.0 migration.

Note that, while there are data model changes, there should be no required
data migrations. The fields and entities being removed will simply remain
as orphans. I confirmed that the removed types (such as the SUNRUSH_ADD
GracePeriodType) are no longer used in production data, and left types
that are still used, e.g. BillingEvent.Flag.LANDRUSH or
HistoryEntry.Type.ALLOCATE.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228752843
2019-01-10 16:23:35 -05:00
mmuller
8bde7285bf Fix travis build with test isolation
Since we seem to be experiencing weird interference between tests on Travis,
run every test in its own JVM until we find a better solution.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228738848
2019-01-10 16:23:35 -05:00
shicong
4866955c76 Unify dependency declarations into one place
This change unified declarations of third party dependencies into
one dependencies.gradle file by reference to this blog
https://medium.com/freelancer-engineering/managing-dependencies-in-multi-project-builds-with-gradle-7626d9c6448d

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228713862
2019-01-10 16:23:35 -05:00
jianglai
ce29e73879 Upgrade to Gradle 5.1
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228520681
2019-01-10 16:23:35 -05:00
Michael Muller
807fe73db4 Fork for every test to verify that this fixes the travis build. 2019-01-09 11:13:48 -05:00
mmuller
9e0b675275 Don't use Files.copy() when decrypting to stdout
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout.  Instead copy directly from the decoder
to standard output.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
2019-01-08 16:51:11 -05:00
shicong
04a495bc99 Fix Gradle build
There were two issues in the Gradle build.

 1. Missing soyToJava conversion for newly added
    google.registry.ui.soy.otesetup.

 2. testSuccess_setAllowedTldsUncached_newTldNotInCache in
    RegistrarTest modified the cache duration to 600 seconds
    which broke other tests as those tests write to the storage
    in initiation stage and expect the data to be available in
    the cache immediately, this requires the duration to be
    0.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228377715
2019-01-08 16:49:39 -05:00
jianglai
c3de49130a Fix Gradle build
Added SOY templates for OTE console need to be complied.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228255528
2019-01-08 11:06:51 -05:00