It looks like I must have somehow duplicated the absolute path when I copied
these, this puts them in the right place (I think...)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233412326
This CL changes the Cloud Build flows to retrieve dependencies from our self-hosted GCS repository, to ensure that the release build are reproducible and hermetic (Note that it is still not truely reproducible as the dependency publishing process will override any existing artifacts in GCS with the current artifacts in Maven central. This is an issue that we should fix later).
There are a couple of changes involved to get this working:
1. Changed internal repo location to pull from the new repo.
2. Remove jcenter repo. It is only used to pull in the docker gradle plugin, which is not used. We instead build the deploy jar file with Gradle and build the docker image with a Dockerfile. The docker gradle plugin artifacts uploaded to GCS cannot be read because it is using some special classifier which seems to not be preserved when uploading. The java application plugin is also removed because it is only used by the docker gradle plugin.
3. Removed netty tcnative library classifier. It does not appear to be actually used (the jar downloaded from Maven central is an uber jar) and the classifier again interferes with downloading the artifacts from GCS.
4. Removed the cyclic dependency of the util project on itself. It was added because the nebula linter wanted it, which I think is an erroneous warning which should be reported upstream. The cyclic dependency was not a problem before (for yet unknown reasons), but it seems like when we force the dependency resolution (by calling project.generateDependencyPublications during configuration stage) it exacerbated the hidden issue and caused a cyclic task dependency in the util project, which is fatal. Now Nebula will complain again, but the warning is considered benign and will not cause the build to fail.
5. Added the nebula dependency lock files. We need these files when using the GCS maven repo because the we only upload artifacts after conflict resolution to GCS. If both v1 and v2 of the same library are requested in the dependency graph, only one will be uploaded. If we do not have the lock files in place, when building from GCS maven repo, Gradle will try to first find both v1 and v2 in the repo (which fails because v1 is not present in the repo), before proceeding to select v2 to use.
6. Refactored the code to upload Maven artifacts to GCS. We need to manually edit the POM file to reproduce the dependencies for each artifact so that they are all put in the classpath during compilation. Before, the POM files do not have any dependency information, which causes compilation to fail because transitive dependencies are not loaded (even though they are present in the GCS repo).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233408051
This allows us to have the source mapping appear when debugging, just like we have when logging on to alpha, prod, or locally.
Tested on Crash with building from Gradle
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233092769
This CL does a few things:
- Adds the template Soy-to-JS compilation (note: this requires the extra
soyutils_usegoog.js file separately so that the compiled *.soy.js files work
- Adds the Closure Compiler to compile and check our JS
- Adds an NPM task to allow us to download dependencies
- Adds the Closure library as an NPM package
Note: this probably won't compile until we fix the test JS files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233059414
This makes such tasks work with Gradle's incremental build.
In practice this change is not very useful:
- The Dagger annotation processor version we use is not fully
compatible with Gradle incremental build, and currently causes
unnecessary rebuild in the util package, and forces tests to always run.
Will try the latest version later, which claims to support incremental
build.
- AppEngine deploy task leaves behind non-writable tools.jar files.
Reruns of any builds that include :services:*:explodeWar would fail,
and the easiest work-around right now is to run Gradle clean task.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232929552
The publish and generateDependencyMetadata tasks need to run
on root project as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232742630
Remove Nebula plugin and enable Gradle locking.
Update publishing task to include build dependencies in resolved artifacts
list.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232349134
We make some modifications to the internal Google checkstyle file because Google's linter uses a modified build of Checkstyle that introduces some new classes and allows for more specific checks than the open-source Checkstyle (e.g. only enforcing UPPER_SNAKE_CASE on deeply immutable fields). There exists a public Checkstyle file that purports to be the Google java format file (https://github.com/checkstyle/checkstyle/blob/master/src/main/resources/google_checks.xml) but it doesn't quite match up with what the internal linter says in certain situations (e.g. what operators must/can appear on new lines).
The suppressions are basically "don't run on generated code + don't care about Javadoc in test code"
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231283700
Identified a few more tests that are impacted by test cleanup
issues outside of Nomulus code base, more likely thread pool
cleanup behaviors in AppEngine testing fixture.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229934266
This console is only to be used by Admins (either GAE admins for this project, or Support accounts). It is for "internal" use only, not for use by the registrars themselves.
To prevent abuse, the registrar is created in a non-functional PENDING state and can only be made functional from the nomulus shell tool.
While in "PENDING" state, the registrar can be updated from the registrar-console by admins.
Also - moving all the web consoles to the same directory (moving the otesetup/* files into registrar/)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229681011
This change fixes the following warnings when applied to cloned
github nomulus:
warning undeclared-dependency one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly
warning undeclared-dependency one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly
warning unused-dependency one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly
warning unused-dependency one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly
✖ 4 problems (0 errors, 4 warnings)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229629444
*** Reason for rollback ***
Found more tests failing.
*** Original change description ***
Reenable Test Executor sharing in Gradle build
Combining all tests in one suite and drop the forkEvery=1 directive.
Issue was fixed by [] and []
TESTED=Run locally with maxParallelForks =1 and 5, and tested on travis
with maxParallelForks=5
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229430425
Combining all tests in one suite and drop the forkEvery=1 directive.
Issue was fixed by [] and []
TESTED=Run locally with maxParallelForks =1 and 5, and tested on travis
with maxParallelForks=5
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229414759
For each registrar, the daily email will only include threats that did not appear
in the prior run's email.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228889972
This backs out most of [] fixes the external build (which wasn't
finding Apache Commons correctly), and makes miscellaneous tweaks and fixes,
including better handling representing the default case of decrypting to stdout.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228877090
This also deletes the associated commands and domain application specific
entities.
We haven't used any of these TLD phases since early 2015 and have no
intent to do so in the future, so it makes sense to delete them now so we
don't have to carry them through the Registry 3.0 migration.
Note that, while there are data model changes, there should be no required
data migrations. The fields and entities being removed will simply remain
as orphans. I confirmed that the removed types (such as the SUNRUSH_ADD
GracePeriodType) are no longer used in production data, and left types
that are still used, e.g. BillingEvent.Flag.LANDRUSH or
HistoryEntry.Type.ALLOCATE.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228752843
Since we seem to be experiencing weird interference between tests on Travis,
run every test in its own JVM until we find a better solution.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228738848
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout. Instead copy directly from the decoder
to standard output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
There were two issues in the Gradle build.
1. Missing soyToJava conversion for newly added
google.registry.ui.soy.otesetup.
2. testSuccess_setAllowedTldsUncached_newTldNotInCache in
RegistrarTest modified the cache duration to 600 seconds
which broke other tests as those tests write to the storage
in initiation stage and expect the data to be available in
the cache immediately, this requires the duration to be
0.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228377715
The artifacts for each service will be packaged inside a tar file that, when untared, is ready for Spinnaker to deploy to GAE.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226516223
Recent changes cause a bunch of new tests to have runtime conflicts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226499557
Added a property to show all test output, useful for debugging.
Usage: ./gradlew -PshowAllOutput=true build
Tested:
Verified that the property does the right thing for true and false values
and verified that the build doesn't show test output if it's omitted.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225854375
The private repo structure is re-done to mirror that of the public repo to facilitate easy merging.
Also removed steps to tag the private repos. This will likely cause a race condition if both the nomulus and the proxy cloud build are triggered by the same tag. They will both try to tag the private repo with the same tag.
The tagging of the private repo should happen simultaneously with the public repo tagging, and in an out-of-band process as far as the build process is concerned. The build process should not have side effect on its source.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225544063
If the user runs "nomulus -e [ENV] login --remote", an URL will be provided, the user then can visit the URL on any machine (not necessary where the command is run) and copy&paste back the authorization code to complete authorization.
This makes it easy to login on machines where local browsers are not easily accessible.
Also upgraded nebula lint version to 10.3.5.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225198700
It'd be nice if we can separate out the tool to its own package and reduce the transitive dependencies that it pulls in. However since the entire core project is a dependency of the tool, it doesn't make any difference as we'd be pulling in core and all its transitive dependencies as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224821931
Removed duplicate build files for each service, and also added two tasks that runs deployment/staging for all services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224359198
We are moving away from using Application Default Credentials generated by "gcloud auth application-default login" in our code base and consolidate on using self-managed credentials provided from AuthModule.
One of the remaining dependencies on the ADCs is from beam pipeline deployment commands, which by default use the ADCs to talk to GCS and upload the jar files and templates. In this CL, we explicitly provide the locally created credential to the Options used in deployments.
Also moved all credential qualifiers to CredentialModule, and removed @AppEngineAdminApiCredential, which is no longer used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224199812
RemoteApiOption has a package-private method that takes a Stream representing the content of a JSON and use a GoogleCredential created from it as its credential. This CL uses reflection to change the access modifier of that method in order to supply a credential stream that is self-managed. This is obviously not ideal and prone to breakage in case the getGoogleCredentialStream method is changed. Unfortunately upstream is not willing to make it public citing the reason that GoogleCredential.fromStream() (which getGoogleCredentialStream uses) is a @Beta annotated function (see https://groups.google.com[]forum/#!searchin/domain-registry-eng/remoteapioptions%7Csort:date/domain-registry-eng/Flsah6skszQ/CySZv2XEBwAJ). However this function is introduced 5 years ago as a public function (b857184bfa). I think at this point it is safe to assume that it is part of the widely used APIs and will not change without sufficient notice.
Note here that RemoteApiOptions creates its own copy of GoogleCredential to be used to call App Engine APIs locally, whereas communications to Nomulus endpoints use the Credential provided in AuthModule. Even though both credentials are created from the same client id, client secret and refresh token (the three elements needed to construct a GoogleCredential this way, see https://github.com/googleapis/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/GoogleCredential.java#L842), their refreshes cycles are independent of each other. I verified that refreshing one of the credential does not invalidate the access token of the other credential, as long as it is not expired yet.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224156131
The version of gradle used in compilation is now controlled by the gradle
wrapper and is centrally managed. This means that everyone will be using the
same version of gradle for build, including IC systems like Travis and Cloud
Build.
The Gradle version for this Gradle wrapper is v5.0.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223856122
Add a first pass add Nomulus GAE deployment to our gradle build scripts. This
attempt could stand some improvement. In particular:
- It currently only deploys to crash, per discussion we will make the
environment a parameter.
- This uses a different import mechanism from the other plugins (via a
"buildscript" section) and it looks like the lint plugin is very similar.
I'll experiment with getting it to work that way.
- There is a lot of redundancy in the deployment scripts for each module, and
given that we have the full power of a general programming language it
should be possible for us to reuse the common parts.
But that said, this is a pretty good first step and I wanted to put it out
there.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223366370
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
This CL adds two ways to build an docker image with gradle:
1) Adds a :proxy:deployJar task that builds an uber jar that contains all runtime dependencies. The jar can then be add to a docker image by calling docker with the added Dockerfile. The base image for this image can be both distroless java or openjdk:alpine.
2) Uses the Gradle distribution plugin to build a distribution tar file that contains all dependencies (as separate jar files) and a run script that sets up the classpath before calling the main class. Then the docker application plugin can build a docker image (with the dockerBuildImage task) using the application tar file. This only works with openjdk:alpline base image as the distroless java image does not contain a shell and therefore the script created by the distribution plugin cannot be launched.
We may later decide to use one of the method and remove the other.
Also adds an outcast test pattern that caused the tests to be flaky.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222145192
It will make it easier later to have the proxy project depend on it, rather
than on the entire core project
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222081994
1. Updated nebula lint version to 10.3.1
2. Do not apply plugins in the root projects that are not needed.
3. Only do linting when a build is successful, so that build failure message are not flushed by linter warnings.
4. Added explicitly stated java source and target version.
5. Moved source sets set up to the root project so that it applies to all subprojects. Currently there is only "core", but we will add at least "proxy" and "util" later, both of which will use mostly the same source sets (but with additional inclusion rules to only build classes in a specific sub folder in the source tree). By putting the set up closure inside the subprojects block in the root script, it can be reused.
6. Rename maybe_runtime configuration to maybeRuntime, which is consistent with other camelCase configuration names like testCompile.
7. Added a runtime dependency of the JAXB API which is removed from Java SE as of version 11.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222081092
These source sets are not configured correctly (no classpaths for example) and
they subsequently cannot compile. The special tests are put into separate tasks that
run from the same source set (test).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222080273
