* Use JSON API for Maven Repo on GCS
The url pattern https://storage.googleapis.com/{Bucket}/{Path}
uses the legacy XML API, which seems to be less robust than
the JSON API. We have observed connection resets after a few
thousand-file download bursts over 30 minutes.
This PR changes all urls to registry's Maven repo on GCS to
gcs://{Bucket}/{Path}. Gradle uses the JSON API for such urls.
TESTED=In Cloud Build with local change
* Use dependency cache in all Gradle tasks in GCB
Make the initial test and the final publishing steps use the shared
dependency cache.
Also make the initial test use the registry's own maven repo instead
of Maven Central.
* Make Gradle dependency cache shareable in GCB
Make Gradle put its caches in the source tree so that
they can be preserved across steps. When left at their
default location, caches are lost after each step.
* Work around Spinnaker issue wrt variables
Cloud Build variable reference need to stay from the ${var} pattern
to prevent Spinnaker from trying to resolve it. In all files that
are used by Spinnaker, we change variable reference to the $var form.
We made the minimum amount of change possible, and will review this
issue after the permanent solution is available.
* Run cross-release SQL integration tests
Run SQL integration tests across arbitrary schema and server
releases.
Refer to integration/README.md in this change for more information.
TESTED=Cloud build changes tested with cloud-build-local
Used the published jars to test sqlIntegration task locally.
* Stop publish Cloud SQL schema jar to maven repo
The original purpose of the maven publication is for
use in server/schema compatibility tests. A commandline
flag can direct a test run to use different versions of
the schema jar. However, this won't work due to dependency
locking.
Defined Docker image for schema deployment.
Included schema deploymer docker in the Cloud Build release process.
Defined Cloud Build config for schema deployment.
TESTED=Used cloud-build-local to test deployment flow
TESTED=Used docker to test schema deployer image in more ways
* Release SQL schema in Cloud Build
Tentatively release SQL schema at the same time as the server release.
Publish schema jar to gs://domain-registry-maven-repository/nomulus
and also upload it with server artifacts.
Also removed the Gradle 'version' variable which is not used.
Tested=On cloud-build with a simplified version of
cloudbuild-nomulus.yaml.
* Save release tag during deployment
* Save current tag for every environment
Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.
* Save current tag for every environment
Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.
* Merge beam and GAE configs deployment to one GCB job
Deployment of GAE configs requires that the credential used by gcloud to
have GAE admin role of the project to be managed. We do not want to
grant the GCB service account that role, because it would all *any* GCB
job to deploy anything to GAE. Instead we use a dedicated credential
originally created to deploy beam pipelines. This credential is
encrypted by KMS and stored on GCS. Since the beam pipeline deployment
GCB job already does the decryption, it make sense to add the config
deployment step there as well. The beam deployment steps are tweaked to
use the nomulus tool docker image instead of the jar file.
Also moved the content of deploy_configs_to_env.sh to the GCB yaml file
itself because the shell script is not uploaded to GC Bat the same time
as the yaml file when the job is triggered by Spinnaker.
Lastly, due to b/137891685, using GCB to deploy cron jobs does not work
as we cannot use service account credential to deploy to projects under
google.com.
* Add a Cloud Build task to update YAML configs
* CR responses
* Move config deployment to a script
* Pin builder version
* Create different beam and deploy-config files per environment
* Update comments and make a for loop
* Build docker image of nomulus tool
In the course of "gradle build", build a docker image of nomulus tool so that
users can run this to allow us to bundle the java version with the image.
* Update GCB beam deployment pipeline
Some of the texts are not really secerts because they are per-project.
Also changed the location of the credential file to `secerts` so that in
the future we may add more secerts in that folder.
The encrypted file is base64 encoded, consistent with how the proxy
certificates are encoded. Also made some changes to the other pipelines
to facilitate automation with Spinnaker
* Move terraform and kubernetes folder to be under proxy
There is no reason for them to be under proxy/src/... any more now that
we have a Gradle-idiomatic folder structure.
With https://github.com/spinnaker/spinnaker/issues/4048 Spinnaker now natively supports GCB. We are able to start a GCB job from Spinnaker, and also there is better support to consume GCB pub/sub messages. Some changes are made to remove the workaround no longer needed.
Two new workflows are added, one to rsync a GCS folder to live/ after the deployment is done (so that the nomulus.jar file can then be fetched to x20 by a []cron job), and the other to tag the proxy image as live once it is deployed.
Lastly, the docs/ folders are needed when running tests. Remove it from .gcloudignore so that when a test run is kicked off by running "gcloud builds submit" the folder is sent to GCB. Ideally .gcloudignore should be identical to .gitignore but since they both are version controlled it is hard it make one a symlink of another.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252625901
We need to first populate the apt sources before installing anything. It was working on my machine because the layer was cached. After I purged all images I was able to replicate the same error message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251918035
This makes it possible to pull the gradle binary from a trusted source when building the release artifacts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=245450562
Refactor out the build and package logic to a reusable script. Also removed the gradle task flag to skip lint check, as failing lint check is no longer a fatal error.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=245296563
This alleviates the needs to have Google credentials even when downloading from
a public bucket.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=245290867
The image name in the manifest needs to be the same as the name that Spinnaker trigger catches. With the new release, Spinnaker now correctly recognizes gcr.io/${PROJECT_ID}/proxy as the image name.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=244845037
The pipeline is broken into two. The first one is to be triggered when the public repo is tagged. It then tags the private repo, builds and upload the builder and base images, and push a new commit to the release (merged repo). This pipeline also does text manipulation on several files in the release repo to ensure that the images uploaded in this pipeline is always used to reproducibly build the release repo at the same commit.
The second pipeline is then triggered by commit into the release repo, which builds, signs and uploads the proxy image.
Also updated the dependency lock files to use the latest plugins dependencies, which are uploaded to the GCS repo.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=244666211
