mirror of
https://github.com/google/nomulus.git
synced 2025-08-06 09:45:19 +02:00
Update proxy deployment pipeline
The pipeline is broken into two. The first one is to be triggered when the public repo is tagged. It then tags the private repo, builds and upload the builder and base images, and push a new commit to the release (merged repo). This pipeline also does text manipulation on several files in the release repo to ensure that the images uploaded in this pipeline is always used to reproducibly build the release repo at the same commit. The second pipeline is then triggered by commit into the release repo, which builds, signs and uploads the proxy image. Also updated the dependency lock files to use the latest plugins dependencies, which are uploaded to the GCS repo. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=244666211
This commit is contained in:
jianglai
parent
7a006df6c3
commit
926e68e806
8 changed files with 181 additions and 81 deletions
25
release/builder/Dockerfile
Normal file
25
release/builder/Dockerfile
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# Copyright 2019 The Nomulus Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# This Dockerfile builds an image that can be used in Google Cloud Build.
|
||||
# We need the following programs to build the Nomulus app and the proxy:
|
||||
# 1. Java 8 for compilation.
|
||||
# 2. Node.js/NPM for JavaScript compilation.
|
||||
# 3. Google Cloud SDK for generating the WARs.
|
||||
# 4. Git to manipulate the private and the merged repos.
|
||||
# 5. Docker to build and push images.
|
||||
FROM marketplace.gcr.io/google/ubuntu1804
|
||||
ENV DEBIAN_FRONTEND=noninteractive LANG=en_US.UTF-8
|
||||
ADD ./build.sh .
|
||||
RUN ["bash", "./build.sh"]
|
||||
43
release/builder/build.sh
Executable file
43
release/builder/build.sh
Executable file
|
|
@ -0,0 +1,43 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2019 The Nomulus Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -e
|
||||
apt-get install locales -y
|
||||
locale-gen en_US.UTF-8
|
||||
apt-get install apt-utils -y
|
||||
apt-get update -y
|
||||
apt-get upgrade -y
|
||||
# Install Java
|
||||
apt-get install openjdk-8-jdk-headless -y
|
||||
# Install npm
|
||||
apt-get install npm -y
|
||||
# Install gcloud
|
||||
# Cribbed from https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu
|
||||
apt-get install lsb-release -y
|
||||
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"
|
||||
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" \
|
||||
| tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
|
||||
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \
|
||||
| apt-key add -
|
||||
apt-get update -y
|
||||
apt-get install google-cloud-sdk-app-engine-java -y
|
||||
# Install git
|
||||
apt-get install git -y
|
||||
# Install docker
|
||||
apt-get install docker.io -y
|
||||
apt-get remove apt-utils locales lsb-release -y
|
||||
apt-get autoclean -y
|
||||
apt-get autoremove -y
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
73
release/cloudbuild-nomulus.yaml
Normal file
73
release/cloudbuild-nomulus.yaml
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
# To run the build locally, install cloud-build-local first.
|
||||
# See: https://cloud.google.com/cloud-build/docs/build-debug-locally
|
||||
# You will need access to a private registry, so be sure to install the docker
|
||||
# credential helper.
|
||||
# Then, in the root of a nomulus source tree, run:
|
||||
# cloud-build-local --config=cloudbuild-nomulus.yaml --dryrun=false --substitutions TAG_NAME=[TAG] .
|
||||
# This will build the contents of the current directory and generate the
|
||||
# nomulus war-files locally.
|
||||
# The PROJECT_ID is the current project name that gcloud uses.
|
||||
# You can add "--push true" to have the image pushed to GCR.
|
||||
#
|
||||
# To manually trigger a build on GCB, run:
|
||||
# gcloud builds submit --config cloudbuild-nomulus.yaml --substitutions TAG_NAME=[TAG] .
|
||||
#
|
||||
# To trigger a build automatically, follow the instructions below and add a trigger:
|
||||
# https://cloud.google.com/cloud-build/docs/running-builds/automate-builds
|
||||
steps:
|
||||
# Set permissions correctly. Not sure why it is necessary, but it is.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: /bin/bash
|
||||
args: ['chown', '-R', 'root:root', '.']
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: /bin/bash
|
||||
args: ['chmod', '-R', '777', '.']
|
||||
# Create a directory to store the artifacts
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: /bin/bash
|
||||
args: ['mkdir', 'nomulus']
|
||||
# Run tests
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args: ['./gradlew', 'test']
|
||||
dir: 'gradle'
|
||||
# Build the deployment files for sandbox.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args:
|
||||
- './gradlew'
|
||||
- 'stage'
|
||||
- '-x'
|
||||
- 'autoLintGradle'
|
||||
- '-Penvironment=sandbox'
|
||||
- '-PmavenUrl=gcs://domain-registry-maven-repository/maven'
|
||||
- '-PpluginsUrl=gcs://domain-registry-maven-repository/plugins'
|
||||
dir: 'gradle'
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: bin/bash
|
||||
args: ['./move_artifacts.sh', 'sandbox', 'nomulus']
|
||||
# Build the deployment files for crash.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args:
|
||||
- './gradlew'
|
||||
- 'stage'
|
||||
- '-x'
|
||||
- 'autoLintGradle'
|
||||
- '-Penvironment=crash'
|
||||
- '-PmavenUrl=gcs://domain-registry-maven-repository/maven'
|
||||
- '-PpluginsUrl=gcs://domain-registry-maven-repository/plugins'
|
||||
dir: 'gradle'
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: bin/bash
|
||||
args: ['./move_artifacts.sh', 'crash', 'nomulus']
|
||||
# Create the uber tarball including all environments.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: bin/bash
|
||||
args: ['tar', 'cvf', '../nomulus.tar', '.']
|
||||
dir: 'nomulus'
|
||||
# The tarball to upload to GCS.
|
||||
artifacts:
|
||||
objects:
|
||||
location: 'gs://${PROJECT_ID}-deploy/${TAG_NAME}'
|
||||
paths: ['nomulus.tar']
|
||||
timeout: 3600s
|
||||
options:
|
||||
machineType: 'N1_HIGHCPU_8'
|
||||
89
release/cloudbuild-proxy.yaml
Normal file
89
release/cloudbuild-proxy.yaml
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
# To run the build locally, install cloud-build-local first.
|
||||
# You will need access to a private registry, so be sure to install the docker
|
||||
# credential helper.
|
||||
# See: https://cloud.google.com/cloud-build/docs/build-debug-locally
|
||||
# Then run:
|
||||
# cloud-build-local --config=cloudbuild-proxy.yaml --dryrun=false --substitutions TAG_NAME=[TAG] ..
|
||||
# This will create a docker image named gcr.io/[PROJECT_ID]/proxy:[TAG] locally.
|
||||
# The PROJECT_ID is the current project name that gcloud uses.
|
||||
#
|
||||
# To manually trigger a build on GCB, run:
|
||||
# gcloud builds submit --config cloudbuild-proxy.yaml --substitutions TAG_NAME=[TAG] ..
|
||||
#
|
||||
# To trigger a build automatically, follow the instructions below and add a trigger:
|
||||
# https://cloud.google.com/cloud-build/docs/running-builds/automate-builds
|
||||
steps:
|
||||
# Set permissions correctly. Not sure why it is necessary, but it is.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args: ['chown', '-R', 'root:root', '.']
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args: ['chmod', '-R', '777', '.']
|
||||
# Build the deploy jar.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args:
|
||||
- './gradlew'
|
||||
- ':proxy:test'
|
||||
- ':proxy:deployJar'
|
||||
- '-x'
|
||||
- 'autoLintGradle'
|
||||
- '-PmavenUrl=gcs://domain-registry-maven-repository/maven'
|
||||
- '-PpluginsUrl=gcs://domain-registry-maven-repository/plugins'
|
||||
dir: 'gradle'
|
||||
# Build the docker image.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args: ['docker', 'build', '--tag', 'gcr.io/${PROJECT_ID}/proxy:${TAG_NAME}', '.']
|
||||
dir: 'gradle/proxy'
|
||||
# Move config files to the working directory. This is necessary because of Spinnaker limitations.
|
||||
# It will concantinate `location' and `path' in the artifact field to construct the artifact
|
||||
# path, even though the artifact is always uploaded to the `location', and `path' can be a regular
|
||||
# expression.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: /bin/bash
|
||||
args: ['-c', 'mv java/google/registry/proxy/kubernetes/* .']
|
||||
# Replace the tag "latest" with the git tag that triggered this build. This is due to a bug in
|
||||
# Spinnaker where the tag is appended to the image name when the deployment pipeline is triggered
|
||||
# by GCB pubsub messages. The bug is fixed in https://github.com/spinnaker/echo/pull/498 and we can
|
||||
# remove this step and the "latest" tag in the manifests when Spinnaker 1.13 is deployed.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: /bin/bash
|
||||
args: ['-c', 'sed -i s/:latest/:${TAG_NAME}/ proxy-*.yaml']
|
||||
# Push the image. We can't let Cloud Build's default processing do that for us
|
||||
# because we need to push the image before we can sign it in the following
|
||||
# step.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
args: ['docker', 'push', 'gcr.io/${PROJECT_ID}/proxy:${TAG_NAME}']
|
||||
# Get the image hash and sign it.
|
||||
- name: 'gcr.io/${PROJECT_ID}/builder:latest'
|
||||
entrypoint: /bin/bash
|
||||
args:
|
||||
- -c
|
||||
- >
|
||||
hash=$(gcloud container images list-tags gcr.io/${PROJECT_ID}/proxy \
|
||||
--format="get(digest)" --filter="tags = ${TAG_NAME}") && \
|
||||
gcloud --project=${PROJECT_ID} alpha container binauthz attestations \
|
||||
sign-and-create --artifact-url=gcr.io/${PROJECT_ID}/proxy@$hash \
|
||||
--attestor=build-attestor --attestor-project=${PROJECT_ID} \
|
||||
--keyversion-project=${PROJECT_ID} --keyversion-location=global \
|
||||
--keyversion-keyring=attestor-keys --keyversion-key=signing \
|
||||
--keyversion=1
|
||||
# Images to upload to GCR. Even though the image has already been uploaded, we still include it
|
||||
# here so that the GCB pubsub message contains it (for Spinnaker to consume).
|
||||
images: ['gcr.io/${PROJECT_ID}/proxy:${TAG_NAME}']
|
||||
# Config files to upload to GCS.
|
||||
artifacts:
|
||||
objects:
|
||||
location: 'gs://${PROJECT_ID}-deploy/${TAG_NAME}'
|
||||
# This cannot be regexs because of how Spinnaker constructs artifact paths.
|
||||
paths:
|
||||
- 'proxy-deployment-alpha.yaml'
|
||||
- 'proxy-deployment-crash.yaml'
|
||||
- 'proxy-deployment-sandbox.yaml'
|
||||
- 'proxy-deployment-production.yaml'
|
||||
- 'proxy-deployment-crash-canary.yaml'
|
||||
- 'proxy-deployment-sandbox-canary.yaml'
|
||||
- 'proxy-deployment-production-canary.yaml'
|
||||
- 'proxy-service.yaml'
|
||||
- 'proxy-service-canary.yaml'
|
||||
timeout: 3600s
|
||||
options:
|
||||
machineType: 'N1_HIGHCPU_8'
|
||||
92
release/cloudbuild-release.yaml
Normal file
92
release/cloudbuild-release.yaml
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
# To run the build locally, install cloud-build-local first.
|
||||
# You will need access to a private registry, so be sure to install the docker
|
||||
# credential helper.
|
||||
# See: https://cloud.google.com/cloud-build/docs/build-debug-locally
|
||||
# Then run:
|
||||
# cloud-build-local --config=cloudbuild-release.yaml --dryrun=false \
|
||||
# --substitutions TAG_NAME=[TAG] ..
|
||||
#
|
||||
# To manually trigger a build on GCB, run:
|
||||
# gcloud builds submit --config cloudbuild-proxy.yaml --substitutions TAG_NAME=[TAG] ..
|
||||
#
|
||||
# To trigger a build automatically, follow the instructions below and add a trigger:
|
||||
# https://cloud.google.com/cloud-build/docs/running-builds/automate-builds
|
||||
#
|
||||
# This pipeline prepares a release. The pipeline should be run against the Nomulus public repo on
|
||||
# GitHub. It builds the builder and base images, and hard codes the sha256 hashes of the resulting
|
||||
# images in the merged code base (internal + public) , which is tagged and pushed into the release
|
||||
# repo. Actual release artifacts are built from the release repo, ensuring reproducibility.
|
||||
steps:
|
||||
# Check the out internal repo.
|
||||
- name: 'gcr.io/cloud-builders/gcloud'
|
||||
args: ['source', 'repos', 'clone', 'nomulus-internal']
|
||||
# Tag and push the internal repo.
|
||||
- name: 'gcr.io/cloud-builders/git'
|
||||
entrypoint: /bin/bash
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
git tag ${TAG_NAME} && git push origin ${TAG_NAME}
|
||||
dir: 'nomulus-internal'
|
||||
# Merge the repos.
|
||||
- name: 'gcr.io/cloud-builders/git'
|
||||
entrypoint: /bin/bash
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
shopt -s dotglob
|
||||
rm -rf .git && rm -rf nomulus-internal/.git
|
||||
cp -rf nomulus-internal/* .
|
||||
rm -rf nomulus-internal
|
||||
# Build the builder image and tag the proxy base image, to be uploaded later.
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
entrypoint: /bin/bash
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
docker build -t gcr.io/${PROJECT_ID}/builder:${TAG_NAME} .
|
||||
docker tag gcr.io/${PROJECT_ID}/builder:${TAG_NAME} gcr.io/${PROJECT_ID}/builder:latest
|
||||
docker pull gcr.io/distroless/java
|
||||
docker tag gcr.io/distroless/java gcr.io/${PROJECT_ID}/base:${TAG_NAME}
|
||||
docker tag gcr.io/distroless/java gcr.io/${PROJECT_ID}/base:latest
|
||||
docker push gcr.io/${PROJECT_ID}/base:latest
|
||||
docker push gcr.io/${PROJECT_ID}/base:${TAG_NAME}
|
||||
docker push gcr.io/${PROJECT_ID}/builder:latest
|
||||
docker push gcr.io/${PROJECT_ID}/builder:${TAG_NAME}
|
||||
dir: 'release/builder/'
|
||||
# Do text replacement in the merged repo, hardcoding image hashes.
|
||||
- name: 'gcr.io/cloud-builders/gcloud'
|
||||
entrypoint: /bin/bash
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
builder_digest=$(gcloud container images list-tags gcr.io/${PROJECT_ID}/builder \
|
||||
--format='get(digest)' --filter='tags = ${TAG_NAME}')
|
||||
base_digest=$(gcloud container images list-tags gcr.io/${PROJECT_ID}/base \
|
||||
--format='get(digest)' --filter='tags = ${TAG_NAME}')
|
||||
sed -i s%distroless/java%${PROJECT_ID}/base@$base_digest% gradle/proxy/Dockerfile
|
||||
sed -i s/builder:latest/builder@$builder_digest/g release/cloudbuild-proxy.yaml
|
||||
sed -i s/builder:latest/builder@$builder_digest/g release/cloudbuild-nomulus.yaml
|
||||
sed -i s/GCP_PROJECT/${PROJECT_ID}/ java/google/registry/proxy/kubernetes/proxy-*.yaml
|
||||
# Check out the release repo.
|
||||
- name: 'gcr.io/cloud-builders/gcloud'
|
||||
args: ['source', 'repos', 'clone', 'nomulus-release']
|
||||
# Tag and check in the release repo.
|
||||
- name: 'gcr.io/cloud-builders/git'
|
||||
entrypoint: /bin/bash
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
cp -rf nomulus-release/.git .
|
||||
rm -rf nomulus-release
|
||||
git config --global user.name "Cloud Build"
|
||||
git config --global user.email \
|
||||
$(gcloud auth list --format='get(account)' --filter=active)
|
||||
git add .
|
||||
git commit -m "Release commit for tag ${TAG_NAME}"
|
||||
git push origin master
|
||||
git tag ${TAG_NAME}
|
||||
git push origin ${TAG_NAME}
|
||||
timeout: 3600s
|
||||
options:
|
||||
machineType: 'N1_HIGHCPU_8'
|
||||
31
release/move_artifacts.sh
Executable file
31
release/move_artifacts.sh
Executable file
|
|
@ -0,0 +1,31 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2019 The Nomulus Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# This script moves GAE artifacts for a given environment to a designated location.
|
||||
|
||||
if [ $# -ne 2 ];
|
||||
then
|
||||
echo "Usage: $0 alpha|crash|sandbox|production destination."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
dest="$2/$1"
|
||||
|
||||
mkdir -p "${dest}"
|
||||
|
||||
for service in default pubapi backend tools
|
||||
do
|
||||
mv gradle/services/"${service}"/build/staged-app "${dest}/${service}"
|
||||
done
|
||||
Loading…
Add table
Add a link
Reference in a new issue