diff --git a/docs/configuration.md b/docs/configuration.md index dade2dc93..108e6b522 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -14,10 +14,8 @@ installation of the project: * Create Google Cloud Storage buckets (see the [Architecture documentation](./architecture.md) for more information). -* Modify `ConfigModule.java` and set project-specific settings such as product - name (see below). -* Copy and edit `ProductionRegistryConfigExample.java` with your - project-specific settings (see below). +* Modify configuration files ("nomulus-config-*.yaml") for all environments + you wish to deploy. ## Environments @@ -74,6 +72,7 @@ description of each option: appEngine: projectId: # Your App Engine project ID toolsServiceUrl: https://tools-dot-PROJECT-ID.appspot.com # Insert your project ID + isLocal: false # Causes saved credentials to be used. gSuite: domainName: # Your G Suite domain name @@ -87,14 +86,10 @@ contains an exhaustive list of all settings to override. From a code perspective, all configuration settings ultimately come through the [`RegistryConfig`][registry-config] class. This includes a Dagger module called -`ConfigModule` that provides injectable configuration options. Some legacy -configuration options that can be changed in this class include timeout lengths -and buffer sizes for various tasks, email addresses and URLs to use for various -services, more Cloud Storage bucket names, and WHOIS disclaimer text. Currently, -in order to configure custom configuration, you need to copy `ConfigModule`, -make changes to it, and include your new version instead of the default one in -all Dagger components. All of these options will be replaced with YAML -configuration settings in the near future. +`ConfigModule` that provides injectable configuration options. While most +configuration options can be changed from within the yaml config file, +certain derived options may still need to be overriden by changing the code in +this module. ## OAuth 2 client id configuration @@ -118,10 +113,10 @@ There are three steps to configuration. ["Credentials" page](https://console.developers.google.com/apis/credentials) in the Developer's Console. Click "Create credentials" and select "OAuth client ID" from the dropdown. In the create credentials window, select an - application type of "Other". After creating the client id, return to the - main Credentials page and click the download icon to the right of the client - id that you just created. This will download a json file called the *client - secret file*. + application type of "Desktop app". After creating the client id, + copy the client id and client secret which are displayed in the popup + window. You may also obtain this information by downloading the json file + for the client id. * **Copy the client secret information to the config file:** The *client secret file* contains both the client ID and the client secret. Copy the @@ -161,15 +156,19 @@ an `InMemoryKeyring` populated with dummy values for all secret keys. This allows the codebase to compile and run, but of course any actions that attempt to connect to external services will fail because none of the keys are real. -To configure a production registry system, you will need to write a replacement -module for `DummyKeyringModule` that loads the credentials in a secure way, and -provides them using either an instance of `InMemoryKeyring` or your own custom -implementation of `Keyring`. You then need to replace all usages of -`DummyKeyringModule` with your own module in all of the per-service components -in which it is referenced. The functions in `PgpHelper` will likely prove useful -for loading keys stored in PGP format into the PGP key classes that you'll need -to provide from `Keyring`, and you can see examples of them in action in -`DummyKeyringModule`. +To configure a production registry system, you will need to either use the +KmsKeyring or write your own replacement module using `DummyKeyringModule` for +guidance. Such a module should provide either an instance of +`InMemoryKeyring` or your own custom implementation of `Keyring`. + +In either case, configure the `keyring` section of the config file with the +appropriate parameters. Use an `activeKeyring` of "KMS" with a project id for +KMS to configure accordingly, for example: + + keyring: + activeKeyring: KMS + kms: + projectId: acme-registry-keys ## Per-TLD configuration @@ -187,3 +186,194 @@ and thus do not require code pushes to update. [app-engine-config]: https://cloud.google.com/appengine/docs/java/configuration-files [default-config]: https://github.com/google/nomulus/blob/master/java/google/registry/config/files/default-config.yaml [registry-config]: https://github.com/google/nomulus/blob/master/java/google/registry/config/RegistryConfig.java + +## Cloud SQL Configuration + +Nomulus is in the process of being ported to Cloud SQL. As such, parts of the +system already require access to Cloud SQL and the necessary configuration +must be applied. + +### Create Postgres Cloud SQL Instance + +You can create a cloud SQL instance using the gcloud command: + + $ gcloud sql instances create nomulus --database-version=POSTGRES_11 \ + --cpu=1 --memory=4G + +Note that for a production instance, you will likely want to be far more +generous with both CPU and memory resources. + +Now get the connection name for the new database: + + $ gcloud sql instances describe nomulus | grep connectionName + connectionName: your-project:us-central1:nomulus + +Copy the connection name into your configuration (see below). + +Now set the password for the default user: + + $ gcloud sql users set-password postgres \ + --instance=nomulus --project=$PROJECT_ID \ + --prompt-for-password + +Store this password somewhere secure. + +Now create database users for the tool and for the backend. First, you'll +need to create a password. This can simply be a sequence of random +characters. Write it to the file `/tmp/server.pass` (we'll use a single +password for the two user accounts here, you are encouraged to use different +passwords for your production systems). Make sure that this file does not +contain a newline after the password. Now create the two user accounts: + + $ gcloud sql users create nomulus --instance=nomulus \ + --project=$PROJECT_ID "--password=`cat /tmp/server.pass`" + $ gcloud sql users create tool --instance=nomulus \ + --project=$PROJECT_ID "--password=`cat /tmp/server.pass`" + +Now enable access to the Cloud SQL admin APIs: + + $ gcloud services enable sqladmin.googleapis.com \ + --project=$PROJECT_ID + +### Installing the Schema + +Google's Nomulus team makes use of Spinnaker-based continuous integration to +perform weekly pushes of both the Nomulus software and the SQL database +schema. Organizations wishing to use the Nomulus software will likely want to +do something similar. However, for purposes of this exercise we will push the +schema from the build system. + +First, download the [Cloud SQL +Proxy](https://cloud.google.com/sql/docs/mysql/sql-proxy). This will allow +you to connect to your database from a local workstation without a lot of +additional configuration. + +Create a service account for use with the proxy: + + $ gcloud iam service-accounts create sql-proxy \ + --project=$PROJECT_ID \ + --description="Service account for use with Cloud SQL Proxy" \ + --display-name="Cloud SQL Proxy" + +Give the service account admin permissions: + + $ gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member=serviceAccount:sql-proxy@$PROJECT_ID.iam.gserviceaccount.com \ + --role=roles/cloudsql.admin + +Create a JSON key for the service account: + + $ gcloud iam service-accounts keys create sql-admin.json \ + --project=$PROJECT_ID \ + --iam-account=sql-proxy@$PROJECT_ID.iam.gserviceaccount.com + +Now start the proxy: + + $ PORT=3306 # Use a different value for this if you like. + $ ./cloud_sql_proxy -credential_file=sql-admin.json \ + -instances=$PROJECT_ID:nomulus=tcp:$PORT + 2020/07/01 12:11:20 current FDs rlimit set to 32768, wanted limit is 8500. Nothing to do here. + 2020/07/01 12:11:20 using credential file for authentication; email=sql-proxy@pproject-id.iam.gserviceaccount.com + 2020/07/01 12:11:20 Listening on 127.0.0.1:3306 for project-id:nomulus + 2020/07/01 12:11:20 Ready for new connections + +Finally, upload the new database schema: + + $ ./nom_build :db:flywayMigrate --dbServer=localhost:$PORT \ + --dbName=postgres --dbUser=nomulus --dbPassword=`cat /tmp/server.pass` + +Now you'll need to give the "tool" user access to all tables. You can do this +either with a locally installed version of PostgreSQL or from the Cloud Shell. +From local postgres, first, with your proxy is still running, connect using +psql. + + $ psql -h localhost -p 3306 postgres nomulus ~/w/nom.admin-docs + Password for user nomulus: + psql (12.2 (Debian 12.2-1+build2), server 11.6) + Type "help" for help. + + postgres=> + +Enter the following command at the postgres prompt: + + GRANT SELECT, INSERT, UPDATE, DELETE + ON ALL TABLES IN SCHEMA public + TO tool; + +From the [Google Cloud Console](https://console.developers.google.com), click +the cloud shell icon in the toolbar (the ">_" icon). You should be able to +connect to your database with gcloud: + + $ gcloud sql connect nomulus --user=nomulus + +From this, you should have a postgres prompt and be able to enter the "GRANT" +command specified above. + +### Cloud KMS + +You'll need to choose a project to host the KMS keyring. Best practice is to +create a separate project with a strict IAM policy. However, you can use your +current project when experimenting. + +You need to create a KMS keyring in the chosen project. The default keyring +name is 'nomulus', though you can override it in the config file. + + $ gcloud kms keyrings create "nomulus" --location "global" \ + --project $KEYS_PROJECT_ID + +Next, you need to create two keys in the keyring: + + $ gcloud kms keys create "cloud-sql-password-string" \ + --project $KEYS_PROJECT_ID \ + --location "global" \ + --keyring "nomulus" \ + --purpose "encryption" + + $ gcloud kms keys create "tools-cloud-sql-password-string" \ + --project $KEYS_PROJECT_ID \ + --location "global" \ + --keyring "nomulus" \ + --purpose "encryption" + +#### Install Cloud SQL Passwords in Nomulus Server + +Use the update_kms_keyring command to upload the Cloud SQL passwords to the +Nomulus server. We'll use the password same set of passwords we specified +above when creating database user accounts. These should currently be stored +in `/tmp/server.pass`. + +Paste the password for the Registry server user to a file, say +/tmp/server.pass. Make sure to avoid any trailing '\n' inserted by the editor. + + $ set ENV=alpha + $ nomulus -e $ENV update_kms_keyring --keyname CLOUD_SQL_PASSWORD \ + --input /tmp/server.pass + +Repeat the steps for the tools sql password: + + $ nomulus -e $ENV update_kms_keyring --keyname TOOLS_CLOUD_SQL_PASSWORD \ + --input /tmp/tools.pass + +Use get_keyring_secret command to verify the data you put in: + + $ nomulus -e alpha -e alpha get_keyring_secret --keyname CLOUD_SQL_PASSWORD + [your password] + $ nomulus -e alpha -e alpha get_keyring_secret --keyname CLOUD_SQL_PASSWORD + [your password] + +#### The Relevant Parts of the Configuration File + + cloudSql: + jdbcUrl: jdbc:postgresql://google/postgres + username: nomulus + instanceConnectionName: THE_NAME_SHOWN_ON_THE_DB_INFO_PAGE + + keyring: + activeKeyring: KMS + kms: + projectId: KEYS_PROJECT_ID + + registryTool: + clientId: TOOLS_OAUTH_CLIENT_ID + clientSecret: TOOLS_OAUTH_SECRET + username: tool diff --git a/docs/first-steps-tutorial.md b/docs/first-steps-tutorial.md index 760974536..514fd1f93 100644 --- a/docs/first-steps-tutorial.md +++ b/docs/first-steps-tutorial.md @@ -18,19 +18,9 @@ Using the `nomulus` admin tool currently requires two additional steps to enable full functionality. These steps should _not_ be done for a production deployment - a suitable solution for production is in progress. -1. Modify the `tools` module `web.xml` file to remove admin-only restrictions. - Look for the `admin` element. Comment - out this element, and redeploy the tools module to your live app. - -2. Set up [application default credentials][app-default-creds] in `gcloud` by - running the following command: - -```shell -$ gcloud beta auth application-default login -Your browser has been opened to visit: -[ ... snip logging in via browser ... ] -You are now logged in as [user@email.tld]. -``` +Modify the `tools` module `web.xml` file to remove admin-only restrictions. +Look for the `admin` element. Comment out +this element, and redeploy the tools module to your live app. [app-default-creds]: https://developers.google.com/identity/protocols/application-default-credentials @@ -42,7 +32,8 @@ it'll never be created for real on the Internet at large. ```shell $ nomulus -e alpha create_tld example --roid_suffix EXAMPLE \ - --initial_tld_state GENERAL_AVAILABILITY --tld_type TEST + --initial_tld_state GENERAL_AVAILABILITY --tld_type TEST \ + --dns_writers VoidDnsWriter [ ... snip confirmation prompt ... ] Perform this command? (y/N): y Updated 1 entities. @@ -64,6 +55,11 @@ Updated 1 entities. it is 8 characters or fewer), such as "EXAMPLE." You can also abbreviate the upper-case TLD name down to 8 characters. Refer to the [gTLD Registry Advisory: Correction of non-compliant ROIDs][roids] for further information. +* `--dns_writers` is the list of DNS writer modules that specify how changes + to domains for the TLD are communicated to actual DNS servers. We use + `VoidDnsWriter` in this case so as to not have to set up DNS. Typically + one might use CloudDnsWriter (for Google Cloud DNS) or implement your own + solution. ## Create a registrar diff --git a/docs/install.md b/docs/install.md index 91835c8d7..262bbab2d 100644 --- a/docs/install.md +++ b/docs/install.md @@ -6,6 +6,7 @@ This document covers the steps necessary to download, build, and deploy Nomulus. You will need the following programs installed on your local machine: + * A recent version of the [Java 11 JDK][java-jdk11]. * [Google App Engine SDK for Java][app-engine-sdk], and configure aliases to to the `gcloud` and `appcfg.sh` utilities (you'll use them a lot). @@ -18,9 +19,9 @@ on other operating systems. ## Download the codebase Start off by using git to download the latest version from the [Nomulus GitHub -page](https://github.com/google/nomulus). In the future we will release tagged -stable versions, but for now, just download `HEAD` of the master branch as -follows: +page](https://github.com/google/nomulus). You may checkout any of the daily +tagged versions (e.g. `nomulus-20200629-RC00`), but in general it is also +safe to simply checkout from HEAD: ```shell $ git clone git@github.com:google/nomulus.git @@ -33,16 +34,20 @@ AUTHORS docs javascript python third_party CONTRIBUTING.md google javatests README.md WORKSPACE ``` -The most important directories are: +Most of the directory tree is organized into gradle sub-projects (see +`settings.gradle` for details). The following other top-level directories are +also defined: -* `docs` -- the documentation (including this install guide) -* `java/google/registry` -- all of the source code of the main project -* `javatests/google/registry` -- all of the tests for the project +* `buildSrc` -- Gradle extensions specific to our local build and release + methodology. +* `config` -- Tools for build and code hygiene. +* `docs` -- The documentation (including this install guide) +* `gradle` -- Configuration and code managed by the gradle build system. +* `java-format` -- The Google java formatter and wrapper scripts to use it + incrementally. * `python` -- Some Python reporting scripts -* `scripts` -- Scripts for configuring development environments - -Everything else, especially `third_party`, contains dependencies that are used -by the project. +* `release` -- Configuration for our continuous integration process. +* `third_party` -- External dependencies. ## Build the codebase @@ -50,48 +55,21 @@ The first step is to build the project, and verify that this completes successfully. This will also download and install dependencies. ```shell -$ bazel --batch build --javacopt="-target 8 -source 8" \ - //java{,tests}/google/registry/... -INFO: Found 584 targets... +$ ./nom_build build +Starting a Gradle Daemon (subsequent builds will be faster) +Plugins: Using default repo... + +> Configure project :buildSrc +Java dependencies: Using Maven central... [ .. snip .. ] -INFO: Elapsed time: 124.433s, Critical Path: 116.92s ``` -There may be some warnings thrown, but if there are no errors, then you can -proceed. The most important build output files from the build are the -[ear](https://en.wikipedia.org/wiki/EAR_\(file_format\)) files: +The `nom_build` script is just a wrapper around `gradlew`. Its main +additional value is that it formalizes the various properties used in the +build as command-line flags. -```shell -$ ls bazel-genfiles/java/google/registry/*.ear -registry_alpha.ear registry.ear registry_sandbox.ear -registry_crash.ear registry_local.ear -``` - -Each `ear` file is a compiled version codebase ready to deploy to App Engine for -a specific environment. By default there are five environments, with the unnamed -one being production. Each `ear` file contains App Engine-specific metadata -files in the `META-INF` directory, as well as three directories for the three -services used in the project, `default`, `backend`, and `tools` (each of these -directories is an unpacked -[war](https://en.wikipedia.org/wiki/WAR_\(file_format\)) file. - -## (Optional) Run the tests - -You can run the tests to verify that all expected functionality succeeds in your -build: - -```shell -$ nice bazel --batch test --javacopt="-target 8 -source 8" \ - //javatests/google/registry/... \ - --local_ram_resources=1000 --local_cpu_resources=3 -Executed 360 out of 360 tests: 360 tests pass. -``` - -**Note:** The tests can be pretty resource intensive, so experiment with -different values of parameters to optimize between low running time and not -slowing down your computer too badly. Refer to the [Bazel User -Manual](https://www.bazel.io/versions/master/docs/bazel-user-manual.html) for -more information. +The "build" command builds all of the code and runs all of the tests. This +will take a while. ## Create an App Engine project @@ -117,51 +95,25 @@ You are now logged in as [user@email.tld]. $ gcloud config set project acme-registry-alpha ``` +Now modify `projects.gradle` with the name of your new project: + +
+// The projects to run your deployment Nomulus application.
+rootProject.ext.projects = ['production': 'your-production-project',
+                            'sandbox'   : 'your-sandbox-project',
+                            'alpha'     : 'acme-registry-alpha',
+                            'crash'     : 'your-crash-project']
+
+ +Next follow the steps in [configuration](./configuration.md) to configure the +complete system or, alternately, read on for an initial deploy in which case +you'll need to deploy again after configuration. + ## Deploy the code to App Engine -One interesting quirk about the App Engine SDK is that it can't use `ear` files -in their packed form; you have to unpack them first, then run `appcfg.sh` -commands on the unpacked contents of the `ear`. So grab the compiled `ear` file -for the alpha environment (it's one of the outputs of the build step earlier), -copy it to another directory, and extract it: +AppEngine deployment with gradle is straightforward: -```shell -$ mkdir /path/to/app-dir/acme-registry-alpha -$ unzip bazel-genfiles/java/google/registry/registry_alpha.ear \ - -d /path/to/app-dir/acme-registry-alpha -$ ls /path/to/app-dir/acme-registry-alpha -backend default META-INF tools -``` - -Now deploy the code to App Engine. We must provide a version string, e.g., live. - -```shell -$ appcfg.sh -A acme-registry-alpha --enable_jar_splitting \ - -V live update /path/to/app-dir/acme-registry-alpha -Reading application configuration data... -Processing module default -Oct 05, 2016 12:16:59 PM com.google.apphosting.utils.config.IndexesXmlReader readConfigXml -INFO: Successfully processed /usr/local/google/home/mcilwain/Code/acme-registry-alpha/./default/WEB-INF/datastore-indexes.xml -Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config -Processing module backend -Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config -Processing module tools -Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config - -Beginning interaction for module default... -0% Created staging directory at: '/tmp/appcfg7185922945263751117.tmp' -5% Scanning for jsp files. -20% Scanning files on local disk. -[ ... snip ... ] -Beginning interaction for module backend... -[ ... snip ... ] -Beginning interaction for module tools... -[ ... snip ... ] -``` - -Note that the `update` command deploys all three services of Nomulus. In the -future, if you've only made changes to a single service, you can save time and -upload just that one using the `-M` flag to specify the service to update. + $ ./nom_build appengineDeploy --environment=alpha To verify successful deployment, visit https://acme-registry-alpha.appspot.com/registrar in your browser (adjusting @@ -171,11 +123,12 @@ you need to configure the system and grant access to your Google account. It's time to go to the next step, configuration. Configuration is handled by editing code, rebuilding the project, and deploying -again. See the [configuration guide](./configuration.md) for more details. Once -you have completed basic configuration (including most critically the project ID -in your copy of `ProductionRegistryConfigExample`), you can rebuild and start -using the `nomulus` tool to create test entities in your newly deployed system. -See the [first steps tutorial](./first-steps-tutorial.md) for more information. +again. See the [configuration guide](./configuration.md) for more details. +Once you have completed basic configuration (including most critically the +project ID, client id and secret in your copy of the `nomulus-config-*.yaml` +files), you can rebuild and start using the `nomulus` tool to create test +entities in your newly deployed system. See the [first steps tutorial](./first-steps-tutorial.md) +for more information. [app-engine-sdk]: https://cloud.google.com/appengine/docs/java/download [java-jdk11]: https://www.oracle.com/java/technologies/javase-downloads.html