Update user-facing documentation (#662)

* Update user-facing documentation

Give our docs a complete overhaul to account for changes in the system,
notably the requirement to configure postgresql.

* Fix dangling sentence.

* Merge branch 'master' into admin-docs

docs/configuration.md

@@ -14,10 +14,8 @@ installation of the project:
 
 *   Create Google Cloud Storage buckets (see the [Architecture
     documentation](./architecture.md) for more information).
-*   Modify `ConfigModule.java` and set project-specific settings such as product
+*   Modify configuration files ("nomulus-config-*.yaml") for all environments
-    name (see below).
+    you wish to deploy.
-*   Copy and edit `ProductionRegistryConfigExample.java` with your
-    project-specific settings (see below).
 
 ## Environments
 
@@ -74,6 +72,7 @@ description of each option:
 appEngine:
   projectId: # Your App Engine project ID
   toolsServiceUrl: https://tools-dot-PROJECT-ID.appspot.com  # Insert your project ID
+  isLocal: false  # Causes saved credentials to be used.
 
 gSuite:
   domainName: # Your G Suite domain name
@@ -87,14 +86,10 @@ contains an exhaustive list of all settings to override.
 
 From a code perspective, all configuration settings ultimately come through the
 [`RegistryConfig`][registry-config] class. This includes a Dagger module called
-`ConfigModule` that provides injectable configuration options. Some legacy
+`ConfigModule` that provides injectable configuration options. While most
-configuration options that can be changed in this class include timeout lengths
+configuration options can be changed from within the yaml config file,
-and buffer sizes for various tasks, email addresses and URLs to use for various
+certain derived options may still need to be overriden by changing the code in
-services, more Cloud Storage bucket names, and WHOIS disclaimer text. Currently,
+this module.
-in order to configure custom configuration, you need to copy `ConfigModule`,
-make changes to it, and include your new version instead of the default one in
-all Dagger components. All of these options will be replaced with YAML
-configuration settings in the near future.
 
 ## OAuth 2 client id configuration
 
@@ -118,10 +113,10 @@ There are three steps to configuration.
     ["Credentials" page](https://console.developers.google.com/apis/credentials)
     in the Developer's Console. Click "Create credentials" and select "OAuth
     client ID" from the dropdown. In the create credentials window, select an
-    application type of "Other". After creating the client id, return to the
+    application type of "Desktop app". After creating the client id,
-    main Credentials page and click the download icon to the right of the client
+    copy the client id and client secret which are displayed in the popup
-    id that you just created. This will download a json file called the *client
+    window.  You may also obtain this information by downloading the json file
-    secret file*.
+    for the client id.
 
 *   **Copy the client secret information to the config file:** The *client
     secret file* contains both the client ID and the client secret. Copy the
@@ -161,15 +156,19 @@ an `InMemoryKeyring` populated with dummy values for all secret keys. This
 allows the codebase to compile and run, but of course any actions that attempt
 to connect to external services will fail because none of the keys are real.
 
-To configure a production registry system, you will need to write a replacement
+To configure a production registry system, you will need to either use the
-module for `DummyKeyringModule` that loads the credentials in a secure way, and
+KmsKeyring or write your own replacement module using `DummyKeyringModule` for
-provides them using either an instance of `InMemoryKeyring` or your own custom
+guidance.  Such a module should provide either an instance of
-implementation of `Keyring`. You then need to replace all usages of
+`InMemoryKeyring` or your own custom implementation of `Keyring`.
-`DummyKeyringModule` with your own module in all of the per-service components
+
-in which it is referenced. The functions in `PgpHelper` will likely prove useful
+In either case, configure the `keyring` section of the config file with the
-for loading keys stored in PGP format into the PGP key classes that you'll need
+appropriate parameters.  Use an `activeKeyring` of "KMS" with a project id for
-to provide from `Keyring`, and you can see examples of them in action in
+KMS to configure accordingly, for example:
-`DummyKeyringModule`.
+
+    keyring:
+      activeKeyring: KMS
+      kms:
+        projectId: acme-registry-keys
 
 ## Per-TLD configuration
 
@@ -187,3 +186,194 @@ and thus do not require code pushes to update.
 [app-engine-config]: https://cloud.google.com/appengine/docs/java/configuration-files
 [default-config]: https://github.com/google/nomulus/blob/master/java/google/registry/config/files/default-config.yaml
 [registry-config]: https://github.com/google/nomulus/blob/master/java/google/registry/config/RegistryConfig.java
+
+## Cloud SQL Configuration
+
+Nomulus is in the process of being ported to Cloud SQL.  As such, parts of the
+system already require access to Cloud SQL and the necessary configuration
+must be applied.
+
+### Create Postgres Cloud SQL Instance
+
+You can create a cloud SQL instance using the gcloud command:
+
+    $ gcloud sql instances create nomulus --database-version=POSTGRES_11 \
+       --cpu=1 --memory=4G
+
+Note that for a production instance, you will likely want to be far more
+generous with both CPU and memory resources.
+
+Now get the connection name for the new database:
+
+    $ gcloud sql instances describe nomulus | grep connectionName
+    connectionName: your-project:us-central1:nomulus
+
+Copy the connection name into your configuration (see below).
+
+Now set the password for the default user:
+
+    $ gcloud sql users set-password postgres \
+        --instance=nomulus --project=$PROJECT_ID \
+        --prompt-for-password
+
+Store this password somewhere secure.
+
+Now create database users for the tool and for the backend.  First, you'll
+need to create a password.  This can simply be a sequence of random
+characters.  Write it to the file `/tmp/server.pass` (we'll use a single
+password for the two user accounts here, you are encouraged to use different
+passwords for your production systems).  Make sure that this file does not
+contain a newline after the password.  Now create the two user accounts:
+
+    $ gcloud sql users create nomulus --instance=nomulus \
+      --project=$PROJECT_ID "--password=`cat /tmp/server.pass`"
+    $ gcloud sql users create tool --instance=nomulus \
+      --project=$PROJECT_ID "--password=`cat /tmp/server.pass`"
+
+Now enable access to the Cloud SQL admin APIs:
+
+    $ gcloud services enable sqladmin.googleapis.com \
+        --project=$PROJECT_ID
+
+### Installing the Schema
+
+Google's Nomulus team makes use of Spinnaker-based continuous integration to
+perform weekly pushes of both the Nomulus software and the SQL database
+schema.  Organizations wishing to use the Nomulus software will likely want to
+do something similar.  However, for purposes of this exercise we will push the
+schema from the build system.
+
+First, download the [Cloud SQL
+Proxy](https://cloud.google.com/sql/docs/mysql/sql-proxy).  This will allow
+you to connect to your database from a local workstation without a lot of
+additional configuration.
+
+Create a service account for use with the proxy:
+
+    $ gcloud iam service-accounts create sql-proxy \
+        --project=$PROJECT_ID \
+        --description="Service account for use with Cloud SQL Proxy" \
+        --display-name="Cloud SQL Proxy"
+
+Give the service account admin permissions:
+
+    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
+        --member=serviceAccount:sql-proxy@$PROJECT_ID.iam.gserviceaccount.com \
+        --role=roles/cloudsql.admin
+
+Create a JSON key for the service account:
+
+    $ gcloud iam service-accounts keys create sql-admin.json \
+        --project=$PROJECT_ID \
+        --iam-account=sql-proxy@$PROJECT_ID.iam.gserviceaccount.com
+
+Now start the proxy:
+
+    $ PORT=3306   # Use a different value for this if you like.
+    $ ./cloud_sql_proxy -credential_file=sql-admin.json \
+        -instances=$PROJECT_ID:nomulus=tcp:$PORT
+    2020/07/01 12:11:20 current FDs rlimit set to 32768, wanted limit is 8500. Nothing to do here.
+    2020/07/01 12:11:20 using credential file for authentication; email=sql-proxy@pproject-id.iam.gserviceaccount.com
+    2020/07/01 12:11:20 Listening on 127.0.0.1:3306 for project-id:nomulus
+    2020/07/01 12:11:20 Ready for new connections
+
+Finally, upload the new database schema:
+
+    $ ./nom_build :db:flywayMigrate --dbServer=localhost:$PORT \
+        --dbName=postgres --dbUser=nomulus --dbPassword=`cat /tmp/server.pass`
+
+Now you'll need to give the "tool" user access to all tables.  You can do this
+either with a locally installed version of PostgreSQL or from the Cloud Shell.
+From local postgres, first, with your proxy is still running, connect using
+psql.
+
+    $ psql -h localhost -p 3306 postgres nomulus                                                                                                                         ~/w/nom.admin-docs
+    Password for user nomulus: <enter the password from /tmp/server.pass>
+    psql (12.2 (Debian 12.2-1+build2), server 11.6)
+    Type "help" for help.
+
+    postgres=>
+
+Enter the following command at the postgres prompt:
+
+    GRANT SELECT, INSERT, UPDATE, DELETE
+    ON ALL TABLES IN SCHEMA public
+    TO tool;
+
+From the [Google Cloud Console](https://console.developers.google.com), click
+the cloud shell icon in the toolbar (the ">_" icon).  You should be able to
+connect to your database with gcloud:
+
+    $ gcloud sql connect nomulus --user=nomulus
+
+From this, you should have a postgres prompt and be able to enter the "GRANT"
+command specified above.
+
+### Cloud KMS
+
+You'll need to choose a project to host the KMS keyring. Best practice is to
+create a separate project with a strict IAM policy. However, you can use your
+current project when experimenting.
+
+You need to create a KMS keyring in the chosen project. The default keyring
+name is 'nomulus', though you can override it in the config file.
+
+    $ gcloud kms keyrings create "nomulus" --location "global" \
+        --project $KEYS_PROJECT_ID
+
+Next, you need to create two keys in the keyring:
+
+    $ gcloud kms keys create "cloud-sql-password-string" \
+        --project $KEYS_PROJECT_ID \
+        --location "global" \
+        --keyring "nomulus" \
+        --purpose "encryption"
+
+    $ gcloud kms keys create "tools-cloud-sql-password-string" \
+        --project $KEYS_PROJECT_ID \
+        --location "global" \
+        --keyring "nomulus" \
+        --purpose "encryption"
+
+#### Install Cloud SQL Passwords in Nomulus Server
+
+Use the update_kms_keyring command to upload the Cloud SQL passwords to the
+Nomulus server.  We'll use the password same set of passwords we specified
+above when creating database user accounts.  These should currently be stored
+in `/tmp/server.pass`.
+
+Paste the password for the Registry server user to a file, say
+/tmp/server.pass. Make sure to avoid any trailing '\n' inserted by the editor.
+
+    $ set ENV=alpha
+    $ nomulus -e $ENV update_kms_keyring --keyname CLOUD_SQL_PASSWORD \
+        --input /tmp/server.pass
+
+Repeat the steps for the tools sql password:
+
+    $ nomulus -e $ENV update_kms_keyring --keyname TOOLS_CLOUD_SQL_PASSWORD \
+        --input /tmp/tools.pass
+
+Use get_keyring_secret command to verify the data you put in:
+
+    $ nomulus -e alpha -e alpha get_keyring_secret --keyname CLOUD_SQL_PASSWORD
+    [your password]
+    $ nomulus -e alpha -e alpha get_keyring_secret --keyname CLOUD_SQL_PASSWORD
+    [your password]
+
+#### The Relevant Parts of the Configuration File
+
+    cloudSql:
+      jdbcUrl: jdbc:postgresql://google/postgres
+      username: nomulus
+      instanceConnectionName: THE_NAME_SHOWN_ON_THE_DB_INFO_PAGE
+
+    keyring:
+      activeKeyring: KMS
+      kms:
+        projectId: KEYS_PROJECT_ID
+
+    registryTool:
+      clientId: TOOLS_OAUTH_CLIENT_ID
+      clientSecret: TOOLS_OAUTH_SECRET
+      username: tool

docs/first-steps-tutorial.md

@@ -18,19 +18,9 @@ Using the `nomulus` admin tool currently requires two additional steps to enable
 full functionality.  These steps should _not_ be done for a production
 deployment - a suitable solution for production is in progress.
 
-1.  Modify the `tools` module `web.xml` file to remove admin-only restrictions.
+Modify the `tools` module `web.xml` file to remove admin-only restrictions.
-    Look for the `<auth-constraint>admin</auth-constraint>` element.  Comment
+Look for the `<auth-constraint>admin</auth-constraint>` element.  Comment out
-    out this element, and redeploy the tools module to your live app.
+this element, and redeploy the tools module to your live app.
-
-2.  Set up [application default credentials][app-default-creds] in `gcloud` by
-    running the following command:
-
-```shell
-$ gcloud beta auth application-default login
-Your browser has been opened to visit:
-[ ... snip logging in via browser ... ]
-You are now logged in as [user@email.tld].
-```
 
 [app-default-creds]: https://developers.google.com/identity/protocols/application-default-credentials
 
@@ -42,7 +32,8 @@ it'll never be created for real on the Internet at large.
 
 ```shell
 $ nomulus -e alpha create_tld example --roid_suffix EXAMPLE \
-  --initial_tld_state GENERAL_AVAILABILITY --tld_type TEST
+  --initial_tld_state GENERAL_AVAILABILITY --tld_type TEST \
+  --dns_writers VoidDnsWriter
 [ ... snip confirmation prompt ... ]
 Perform this command? (y/N): y
 Updated 1 entities.
@@ -64,6 +55,11 @@ Updated 1 entities.
     it is 8 characters or fewer), such as "EXAMPLE." You can also abbreviate the
     upper-case TLD name down to 8 characters. Refer to the [gTLD Registry
     Advisory: Correction of non-compliant ROIDs][roids] for further information.
+*   `--dns_writers` is the list of DNS writer modules that specify how changes
+    to domains for the TLD are communicated to actual DNS servers.  We use
+    `VoidDnsWriter` in this case so as to not have to set up DNS.  Typically
+    one might use CloudDnsWriter (for Google Cloud DNS) or implement your own
+    solution.
 
 ## Create a registrar
 

docs/install.md

@@ -6,6 +6,7 @@ This document covers the steps necessary to download, build, and deploy Nomulus.
 
 You will need the following programs installed on your local machine:
 
+
 *   A recent version of the [Java 11 JDK][java-jdk11].
 *   [Google App Engine SDK for Java][app-engine-sdk], and configure aliases to
     to the `gcloud` and `appcfg.sh` utilities (you'll use them a lot).
@@ -18,9 +19,9 @@ on other operating systems.
 ## Download the codebase
 
 Start off by using git to download the latest version from the [Nomulus GitHub
-page](https://github.com/google/nomulus). In the future we will release tagged
+page](https://github.com/google/nomulus). You may checkout any of the daily
-stable versions, but for now, just download `HEAD` of the master branch as
+tagged versions (e.g. `nomulus-20200629-RC00`), but in general it is also
-follows:
+safe to simply checkout from HEAD:
 
 ```shell
 $ git clone git@github.com:google/nomulus.git
@@ -33,16 +34,20 @@ AUTHORS          docs          javascript  python     third_party
 CONTRIBUTING.md  google        javatests   README.md  WORKSPACE
 ```
 
-The most important directories are:
+Most of the directory tree is organized into gradle sub-projects (see
+`settings.gradle` for details).  The following other top-level directories are
+also defined:
 
-*   `docs` -- the documentation (including this install guide)
+*   `buildSrc` -- Gradle extensions specific to our local build and release
-*   `java/google/registry` -- all of the source code of the main project
+    methodology.
-*   `javatests/google/registry` -- all of the tests for the project
+*   `config` -- Tools for build and code hygiene.
+*   `docs` -- The documentation (including this install guide)
+*   `gradle` -- Configuration and code managed by the gradle build system.
+*   `java-format` -- The Google java formatter and wrapper scripts to use it
+    incrementally.
 *   `python` -- Some Python reporting scripts
-*   `scripts` -- Scripts for configuring development environments
+*   `release` -- Configuration for our continuous integration process.
-
+*   `third_party` -- External dependencies.
-Everything else, especially `third_party`, contains dependencies that are used
-by the project.
 
 ## Build the codebase
 
@@ -50,48 +55,21 @@ The first step is to build the project, and verify that this completes
 successfully. This will also download and install dependencies.
 
 ```shell
-$ bazel --batch build --javacopt="-target 8 -source 8" \
+$ ./nom_build build
-  //java{,tests}/google/registry/...
+Starting a Gradle Daemon (subsequent builds will be faster)
-INFO: Found 584 targets...
+Plugins: Using default repo...
+
+> Configure project :buildSrc
+Java dependencies: Using Maven central...
 [ .. snip .. ]
-INFO: Elapsed time: 124.433s, Critical Path: 116.92s
 ```
 
-There may be some warnings thrown, but if there are no errors, then you can
+The `nom_build` script is just a wrapper around `gradlew`.  Its main
-proceed. The most important build output files from the build are the
+additional value is that it formalizes the various properties used in the
-[ear](https://en.wikipedia.org/wiki/EAR_\(file_format\)) files:
+build as command-line flags.
 
-```shell
+The "build" command builds all of the code and runs all of the tests.  This
-$ ls bazel-genfiles/java/google/registry/*.ear
+will take a while.
-registry_alpha.ear  registry.ear        registry_sandbox.ear
-registry_crash.ear  registry_local.ear
-```
-
-Each `ear` file is a compiled version codebase ready to deploy to App Engine for
-a specific environment. By default there are five environments, with the unnamed
-one being production. Each `ear` file contains App Engine-specific metadata
-files in the `META-INF` directory, as well as three directories for the three
-services used in the project, `default`, `backend`, and `tools` (each of these
-directories is an unpacked
-[war](https://en.wikipedia.org/wiki/WAR_\(file_format\)) file.
-
-## (Optional) Run the tests
-
-You can run the tests to verify that all expected functionality succeeds in your
-build:
-
-```shell
-$ nice bazel --batch test  --javacopt="-target 8 -source 8" \
-  //javatests/google/registry/... \
-  --local_ram_resources=1000 --local_cpu_resources=3
-Executed 360 out of 360 tests: 360 tests pass.
-```
-
-**Note:** The tests can be pretty resource intensive, so experiment with
-different values of parameters to optimize between low running time and not
-slowing down your computer too badly. Refer to the [Bazel User
-Manual](https://www.bazel.io/versions/master/docs/bazel-user-manual.html) for
-more information.
 
 ## Create an App Engine project
 
@@ -117,51 +95,25 @@ You are now logged in as [user@email.tld].
 $ gcloud config set project acme-registry-alpha
 ```
 
+Now modify `projects.gradle` with the name of your new project:
+
+<pre>
+// The projects to run your deployment Nomulus application.
+rootProject.ext.projects = ['production': 'your-production-project',
+                            'sandbox'   : 'your-sandbox-project',
+                            'alpha'     : <strong>'acme-registry-alpha',</strong>
+                            'crash'     : 'your-crash-project']
+</pre>
+
+Next follow the steps in [configuration](./configuration.md) to configure the
+complete system or, alternately, read on for an initial deploy in which case
+you'll need to deploy again after configuration.
+
 ## Deploy the code to App Engine
 
-One interesting quirk about the App Engine SDK is that it can't use `ear` files
+AppEngine deployment with gradle is straightforward:
-in their packed form; you have to unpack them first, then run `appcfg.sh`
-commands on the unpacked contents of the `ear`. So grab the compiled `ear` file
-for the alpha environment (it's one of the outputs of the build step earlier),
-copy it to another directory, and extract it:
 
-```shell
+    $ ./nom_build appengineDeploy --environment=alpha
-$ mkdir /path/to/app-dir/acme-registry-alpha
-$ unzip bazel-genfiles/java/google/registry/registry_alpha.ear \
-  -d /path/to/app-dir/acme-registry-alpha
-$ ls /path/to/app-dir/acme-registry-alpha
-backend  default  META-INF  tools
-```
-
-Now deploy the code to App Engine. We must provide a version string, e.g., live.
-
-```shell
-$ appcfg.sh -A acme-registry-alpha --enable_jar_splitting \
-  -V live update /path/to/app-dir/acme-registry-alpha
-Reading application configuration data...
-Processing module default
-Oct 05, 2016 12:16:59 PM com.google.apphosting.utils.config.IndexesXmlReader readConfigXml
-INFO: Successfully processed /usr/local/google/home/mcilwain/Code/acme-registry-alpha/./default/WEB-INF/datastore-indexes.xml
-Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config
-Processing module backend
-Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config
-Processing module tools
-Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config
-
-Beginning interaction for module default...
-0% Created staging directory at: '/tmp/appcfg7185922945263751117.tmp'
-5% Scanning for jsp files.
-20% Scanning files on local disk.
-[ ... snip ... ]
-Beginning interaction for module backend...
-[ ... snip ... ]
-Beginning interaction for module tools...
-[ ... snip ... ]
-```
-
-Note that the `update` command deploys all three services of Nomulus. In the
-future, if you've only made changes to a single service, you can save time and
-upload just that one using the `-M` flag to specify the service to update.
 
 To verify successful deployment, visit
 https://acme-registry-alpha.appspot.com/registrar in your browser (adjusting
@@ -171,11 +123,12 @@ you need to configure the system and grant access to your Google account. It's
 time to go to the next step, configuration.
 
 Configuration is handled by editing code, rebuilding the project, and deploying
-again. See the [configuration guide](./configuration.md) for more details. Once
+again. See the [configuration guide](./configuration.md) for more details.
-you have completed basic configuration (including most critically the project ID
+Once you have completed basic configuration (including most critically the
-in your copy of `ProductionRegistryConfigExample`), you can rebuild and start
+project ID, client id and secret in your copy of the `nomulus-config-*.yaml`
-using the `nomulus` tool to create test entities in your newly deployed system.
+files), you can rebuild and start using the `nomulus` tool to create test
-See the [first steps tutorial](./first-steps-tutorial.md) for more information.
+entities in your newly deployed system. See the [first steps tutorial](./first-steps-tutorial.md)
+for more information.
 
 [app-engine-sdk]: https://cloud.google.com/appengine/docs/java/download
 [java-jdk11]: https://www.oracle.com/java/technologies/javase-downloads.html