zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-04-29 19:47:51 +02:00
Code Issues Projects Releases Packages Wiki Activity

Update user-facing documentation (#662)

Browse source 
* Update user-facing documentation

Give our docs a complete overhaul to account for changes in the system,
notably the requirement to configure postgresql.

* Fix dangling sentence.

* Merge branch 'master' into admin-docs
This commit is contained in:
Michael Muller 2020-08-17 14:46:32 -04:00 • committed by GitHub
parent 6a5dbf8817
commit d0144290bb
3 changed files with 274 additions and 135 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

240
docs/configuration.md
View file

@ -14,10 +14,8 @@ installation of the project:
* Create Google Cloud Storage buckets (see the [Architecture
documentation](./architecture.md) for more information).
* Modify `ConfigModule.java` and set project-specific settings such as product
name (see below).
* Copy and edit `ProductionRegistryConfigExample.java` with your
project-specific settings (see below).
* Modify configuration files ("nomulus-config-*.yaml") for all environments
you wish to deploy.
## Environments
@ -74,6 +72,7 @@ description of each option:
appEngine:
projectId: # Your App Engine project ID
toolsServiceUrl: https://tools-dot-PROJECT-ID.appspot.com # Insert your project ID
isLocal: false # Causes saved credentials to be used.
gSuite:
domainName: # Your G Suite domain name
@ -87,14 +86,10 @@ contains an exhaustive list of all settings to override.
From a code perspective, all configuration settings ultimately come through the
[`RegistryConfig`][registry-config] class. This includes a Dagger module called
`ConfigModule` that provides injectable configuration options. Some legacy
configuration options that can be changed in this class include timeout lengths
and buffer sizes for various tasks, email addresses and URLs to use for various
services, more Cloud Storage bucket names, and WHOIS disclaimer text. Currently,
in order to configure custom configuration, you need to copy `ConfigModule`,
make changes to it, and include your new version instead of the default one in
all Dagger components. All of these options will be replaced with YAML
configuration settings in the near future.
`ConfigModule` that provides injectable configuration options. While most
configuration options can be changed from within the yaml config file,
certain derived options may still need to be overriden by changing the code in
this module.
## OAuth 2 client id configuration
@ -118,10 +113,10 @@ There are three steps to configuration.
["Credentials" page](https://console.developers.google.com/apis/credentials)
in the Developer's Console. Click "Create credentials" and select "OAuth
client ID" from the dropdown. In the create credentials window, select an
application type of "Other". After creating the client id, return to the
main Credentials page and click the download icon to the right of the client
id that you just created. This will download a json file called the *client
secret file*.
application type of "Desktop app". After creating the client id,
copy the client id and client secret which are displayed in the popup
window. You may also obtain this information by downloading the json file
for the client id.
* **Copy the client secret information to the config file:** The *client
secret file* contains both the client ID and the client secret. Copy the
@ -161,15 +156,19 @@ an `InMemoryKeyring` populated with dummy values for all secret keys. This
allows the codebase to compile and run, but of course any actions that attempt
to connect to external services will fail because none of the keys are real.
To configure a production registry system, you will need to write a replacement
module for `DummyKeyringModule` that loads the credentials in a secure way, and
provides them using either an instance of `InMemoryKeyring` or your own custom
implementation of `Keyring`. You then need to replace all usages of
`DummyKeyringModule` with your own module in all of the per-service components
in which it is referenced. The functions in `PgpHelper` will likely prove useful
for loading keys stored in PGP format into the PGP key classes that you'll need
to provide from `Keyring`, and you can see examples of them in action in
`DummyKeyringModule`.
To configure a production registry system, you will need to either use the
KmsKeyring or write your own replacement module using `DummyKeyringModule` for
guidance. Such a module should provide either an instance of
`InMemoryKeyring` or your own custom implementation of `Keyring`.
In either case, configure the `keyring` section of the config file with the
appropriate parameters. Use an `activeKeyring` of "KMS" with a project id for
KMS to configure accordingly, for example:
keyring:
activeKeyring: KMS
kms:
projectId: acme-registry-keys
## Per-TLD configuration
@ -187,3 +186,194 @@ and thus do not require code pushes to update.
[app-engine-config]: https://cloud.google.com/appengine/docs/java/configuration-files
[default-config]: https://github.com/google/nomulus/blob/master/java/google/registry/config/files/default-config.yaml
[registry-config]: https://github.com/google/nomulus/blob/master/java/google/registry/config/RegistryConfig.java
## Cloud SQL Configuration
Nomulus is in the process of being ported to Cloud SQL. As such, parts of the
system already require access to Cloud SQL and the necessary configuration
must be applied.
### Create Postgres Cloud SQL Instance
You can create a cloud SQL instance using the gcloud command:
$ gcloud sql instances create nomulus --database-version=POSTGRES_11 \
--cpu=1 --memory=4G
Note that for a production instance, you will likely want to be far more
generous with both CPU and memory resources.
Now get the connection name for the new database:
$ gcloud sql instances describe nomulus | grep connectionName
connectionName: your-project:us-central1:nomulus
Copy the connection name into your configuration (see below).
Now set the password for the default user:
$ gcloud sql users set-password postgres \
--instance=nomulus --project=$PROJECT_ID \
--prompt-for-password
Store this password somewhere secure.
Now create database users for the tool and for the backend. First, you'll
need to create a password. This can simply be a sequence of random
characters. Write it to the file `/tmp/server.pass` (we'll use a single
password for the two user accounts here, you are encouraged to use different
passwords for your production systems). Make sure that this file does not
contain a newline after the password. Now create the two user accounts:
$ gcloud sql users create nomulus --instance=nomulus \
--project=$PROJECT_ID "--password=`cat /tmp/server.pass`"
$ gcloud sql users create tool --instance=nomulus \
--project=$PROJECT_ID "--password=`cat /tmp/server.pass`"
Now enable access to the Cloud SQL admin APIs:
$ gcloud services enable sqladmin.googleapis.com \
--project=$PROJECT_ID
### Installing the Schema
Google's Nomulus team makes use of Spinnaker-based continuous integration to
perform weekly pushes of both the Nomulus software and the SQL database
schema. Organizations wishing to use the Nomulus software will likely want to
do something similar. However, for purposes of this exercise we will push the
schema from the build system.
First, download the [Cloud SQL
Proxy](https://cloud.google.com/sql/docs/mysql/sql-proxy). This will allow
you to connect to your database from a local workstation without a lot of
additional configuration.
Create a service account for use with the proxy:
$ gcloud iam service-accounts create sql-proxy \
--project=$PROJECT_ID \
--description="Service account for use with Cloud SQL Proxy" \
--display-name="Cloud SQL Proxy"
Give the service account admin permissions:
$ gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:sql-proxy@$PROJECT_ID.iam.gserviceaccount.com \
--role=roles/cloudsql.admin
Create a JSON key for the service account:
$ gcloud iam service-accounts keys create sql-admin.json \
--project=$PROJECT_ID \
--iam-account=sql-proxy@$PROJECT_ID.iam.gserviceaccount.com
Now start the proxy:
$ PORT=3306 # Use a different value for this if you like.
$ ./cloud_sql_proxy -credential_file=sql-admin.json \
-instances=$PROJECT_ID:nomulus=tcp:$PORT
2020/07/01 12:11:20 current FDs rlimit set to 32768, wanted limit is 8500. Nothing to do here.
2020/07/01 12:11:20 using credential file for authentication; email=sql-proxy@pproject-id.iam.gserviceaccount.com
2020/07/01 12:11:20 Listening on 127.0.0.1:3306 for project-id:nomulus
2020/07/01 12:11:20 Ready for new connections
Finally, upload the new database schema:
$ ./nom_build :db:flywayMigrate --dbServer=localhost:$PORT \
--dbName=postgres --dbUser=nomulus --dbPassword=`cat /tmp/server.pass`
Now you'll need to give the "tool" user access to all tables. You can do this
either with a locally installed version of PostgreSQL or from the Cloud Shell.
From local postgres, first, with your proxy is still running, connect using
psql.
$ psql -h localhost -p 3306 postgres nomulus ~/w/nom.admin-docs
Password for user nomulus: <enter the password from /tmp/server.pass>
psql (12.2 (Debian 12.2-1+build2), server 11.6)
Type "help" for help.
postgres=>
Enter the following command at the postgres prompt:
GRANT SELECT, INSERT, UPDATE, DELETE
ON ALL TABLES IN SCHEMA public
TO tool;
From the [Google Cloud Console](https://console.developers.google.com), click
the cloud shell icon in the toolbar (the ">_" icon). You should be able to
connect to your database with gcloud:
$ gcloud sql connect nomulus --user=nomulus
From this, you should have a postgres prompt and be able to enter the "GRANT"
command specified above.
### Cloud KMS
You'll need to choose a project to host the KMS keyring. Best practice is to
create a separate project with a strict IAM policy. However, you can use your
current project when experimenting.
You need to create a KMS keyring in the chosen project. The default keyring
name is 'nomulus', though you can override it in the config file.
$ gcloud kms keyrings create "nomulus" --location "global" \
--project $KEYS_PROJECT_ID
Next, you need to create two keys in the keyring:
$ gcloud kms keys create "cloud-sql-password-string" \
--project $KEYS_PROJECT_ID \
--location "global" \
--keyring "nomulus" \
--purpose "encryption"
$ gcloud kms keys create "tools-cloud-sql-password-string" \
--project $KEYS_PROJECT_ID \
--location "global" \
--keyring "nomulus" \
--purpose "encryption"
#### Install Cloud SQL Passwords in Nomulus Server
Use the update_kms_keyring command to upload the Cloud SQL passwords to the
Nomulus server. We'll use the password same set of passwords we specified
above when creating database user accounts. These should currently be stored
in `/tmp/server.pass`.
Paste the password for the Registry server user to a file, say
/tmp/server.pass. Make sure to avoid any trailing '\n' inserted by the editor.
$ set ENV=alpha
$ nomulus -e $ENV update_kms_keyring --keyname CLOUD_SQL_PASSWORD \
--input /tmp/server.pass
Repeat the steps for the tools sql password:
$ nomulus -e $ENV update_kms_keyring --keyname TOOLS_CLOUD_SQL_PASSWORD \
--input /tmp/tools.pass
Use get_keyring_secret command to verify the data you put in:
$ nomulus -e alpha -e alpha get_keyring_secret --keyname CLOUD_SQL_PASSWORD
[your password]
$ nomulus -e alpha -e alpha get_keyring_secret --keyname CLOUD_SQL_PASSWORD
[your password]
#### The Relevant Parts of the Configuration File
cloudSql:
jdbcUrl: jdbc:postgresql://google/postgres
username: nomulus
instanceConnectionName: THE_NAME_SHOWN_ON_THE_DB_INFO_PAGE
keyring:
activeKeyring: KMS
kms:
projectId: KEYS_PROJECT_ID
registryTool:
clientId: TOOLS_OAUTH_CLIENT_ID
clientSecret: TOOLS_OAUTH_SECRET
username: tool

24
docs/first-steps-tutorial.md
View file

@ -18,19 +18,9 @@ Using the `nomulus` admin tool currently requires two additional steps to enable
full functionality. These steps should _not_ be done for a production
deployment - a suitable solution for production is in progress.
1. Modify the `tools` module `web.xml` file to remove admin-only restrictions.
Look for the `<auth-constraint>admin</auth-constraint>` element. Comment
out this element, and redeploy the tools module to your live app.
2. Set up [application default credentials][app-default-creds] in `gcloud` by
running the following command:
```shell
$ gcloud beta auth application-default login
Your browser has been opened to visit:
[ ... snip logging in via browser ... ]
You are now logged in as [user@email.tld].
```
Modify the `tools` module `web.xml` file to remove admin-only restrictions.
Look for the `<auth-constraint>admin</auth-constraint>` element. Comment out
this element, and redeploy the tools module to your live app.
[app-default-creds]: https://developers.google.com/identity/protocols/application-default-credentials
@ -42,7 +32,8 @@ it'll never be created for real on the Internet at large.
```shell
$ nomulus -e alpha create_tld example --roid_suffix EXAMPLE \
--initial_tld_state GENERAL_AVAILABILITY --tld_type TEST
--initial_tld_state GENERAL_AVAILABILITY --tld_type TEST \
--dns_writers VoidDnsWriter
[ ... snip confirmation prompt ... ]
Perform this command? (y/N): y
Updated 1 entities.
@ -64,6 +55,11 @@ Updated 1 entities.
it is 8 characters or fewer), such as "EXAMPLE." You can also abbreviate the
upper-case TLD name down to 8 characters. Refer to the [gTLD Registry
Advisory: Correction of non-compliant ROIDs][roids] for further information.
* `--dns_writers` is the list of DNS writer modules that specify how changes
to domains for the TLD are communicated to actual DNS servers. We use
`VoidDnsWriter` in this case so as to not have to set up DNS. Typically
one might use CloudDnsWriter (for Google Cloud DNS) or implement your own
solution.
## Create a registrar

145
docs/install.md
View file

@ -6,6 +6,7 @@ This document covers the steps necessary to download, build, and deploy Nomulus.
You will need the following programs installed on your local machine:
* A recent version of the [Java 11 JDK][java-jdk11].
* [Google App Engine SDK for Java][app-engine-sdk], and configure aliases to
to the `gcloud` and `appcfg.sh` utilities (you'll use them a lot).
@ -18,9 +19,9 @@ on other operating systems.
## Download the codebase
Start off by using git to download the latest version from the [Nomulus GitHub
page](https://github.com/google/nomulus). In the future we will release tagged
stable versions, but for now, just download `HEAD` of the master branch as
follows:
page](https://github.com/google/nomulus). You may checkout any of the daily
tagged versions (e.g. `nomulus-20200629-RC00`), but in general it is also
safe to simply checkout from HEAD:
```shell
$ git clone git@github.com:google/nomulus.git
@ -33,16 +34,20 @@ AUTHORS docs javascript python third_party
CONTRIBUTING.md google javatests README.md WORKSPACE
```
The most important directories are:
Most of the directory tree is organized into gradle sub-projects (see
`settings.gradle` for details). The following other top-level directories are
also defined:
* `docs` -- the documentation (including this install guide)
* `java/google/registry` -- all of the source code of the main project
* `javatests/google/registry` -- all of the tests for the project
* `buildSrc` -- Gradle extensions specific to our local build and release
methodology.
* `config` -- Tools for build and code hygiene.
* `docs` -- The documentation (including this install guide)
* `gradle` -- Configuration and code managed by the gradle build system.
* `java-format` -- The Google java formatter and wrapper scripts to use it
incrementally.
* `python` -- Some Python reporting scripts
* `scripts` -- Scripts for configuring development environments
Everything else, especially `third_party`, contains dependencies that are used
by the project.
* `release` -- Configuration for our continuous integration process.
* `third_party` -- External dependencies.
## Build the codebase
@ -50,48 +55,21 @@ The first step is to build the project, and verify that this completes
successfully. This will also download and install dependencies.
```shell
$ bazel --batch build --javacopt="-target 8 -source 8" \
//java{,tests}/google/registry/...
INFO: Found 584 targets...
$ ./nom_build build
Starting a Gradle Daemon (subsequent builds will be faster)
Plugins: Using default repo...
> Configure project :buildSrc
Java dependencies: Using Maven central...
[ .. snip .. ]
INFO: Elapsed time: 124.433s, Critical Path: 116.92s
```
There may be some warnings thrown, but if there are no errors, then you can
proceed. The most important build output files from the build are the
[ear](https://en.wikipedia.org/wiki/EAR_\(file_format\)) files:
The `nom_build` script is just a wrapper around `gradlew`. Its main
additional value is that it formalizes the various properties used in the
build as command-line flags.
```shell
$ ls bazel-genfiles/java/google/registry/*.ear
registry_alpha.ear registry.ear registry_sandbox.ear
registry_crash.ear registry_local.ear
```
Each `ear` file is a compiled version codebase ready to deploy to App Engine for
a specific environment. By default there are five environments, with the unnamed
one being production. Each `ear` file contains App Engine-specific metadata
files in the `META-INF` directory, as well as three directories for the three
services used in the project, `default`, `backend`, and `tools` (each of these
directories is an unpacked
[war](https://en.wikipedia.org/wiki/WAR_\(file_format\)) file.
## (Optional) Run the tests
You can run the tests to verify that all expected functionality succeeds in your
build:
```shell
$ nice bazel --batch test --javacopt="-target 8 -source 8" \
//javatests/google/registry/... \
--local_ram_resources=1000 --local_cpu_resources=3
Executed 360 out of 360 tests: 360 tests pass.
```
**Note:** The tests can be pretty resource intensive, so experiment with
different values of parameters to optimize between low running time and not
slowing down your computer too badly. Refer to the [Bazel User
Manual](https://www.bazel.io/versions/master/docs/bazel-user-manual.html) for
more information.
The "build" command builds all of the code and runs all of the tests. This
will take a while.
## Create an App Engine project
@ -117,51 +95,25 @@ You are now logged in as [user@email.tld].
$ gcloud config set project acme-registry-alpha
```
Now modify `projects.gradle` with the name of your new project:
<pre>
// The projects to run your deployment Nomulus application.
rootProject.ext.projects = ['production': 'your-production-project',
'sandbox' : 'your-sandbox-project',
'alpha' : <strong>'acme-registry-alpha',</strong>
'crash' : 'your-crash-project']
</pre>
Next follow the steps in [configuration](./configuration.md) to configure the
complete system or, alternately, read on for an initial deploy in which case
you'll need to deploy again after configuration.
## Deploy the code to App Engine
One interesting quirk about the App Engine SDK is that it can't use `ear` files
in their packed form; you have to unpack them first, then run `appcfg.sh`
commands on the unpacked contents of the `ear`. So grab the compiled `ear` file
for the alpha environment (it's one of the outputs of the build step earlier),
copy it to another directory, and extract it:
AppEngine deployment with gradle is straightforward:
```shell
$ mkdir /path/to/app-dir/acme-registry-alpha
$ unzip bazel-genfiles/java/google/registry/registry_alpha.ear \
-d /path/to/app-dir/acme-registry-alpha
$ ls /path/to/app-dir/acme-registry-alpha
backend default META-INF tools
```
Now deploy the code to App Engine. We must provide a version string, e.g., live.
```shell
$ appcfg.sh -A acme-registry-alpha --enable_jar_splitting \
-V live update /path/to/app-dir/acme-registry-alpha
Reading application configuration data...
Processing module default
Oct 05, 2016 12:16:59 PM com.google.apphosting.utils.config.IndexesXmlReader readConfigXml
INFO: Successfully processed /usr/local/google/home/mcilwain/Code/acme-registry-alpha/./default/WEB-INF/datastore-indexes.xml
Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config
Processing module backend
Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config
Processing module tools
Ignoring application.xml context-root element, for details see https://developers.google.com/appengine/docs/java/modules/#config
Beginning interaction for module default...
0% Created staging directory at: '/tmp/appcfg7185922945263751117.tmp'
5% Scanning for jsp files.
20% Scanning files on local disk.
[ ... snip ... ]
Beginning interaction for module backend...
[ ... snip ... ]
Beginning interaction for module tools...
[ ... snip ... ]
```
Note that the `update` command deploys all three services of Nomulus. In the
future, if you've only made changes to a single service, you can save time and
upload just that one using the `-M` flag to specify the service to update.
$ ./nom_build appengineDeploy --environment=alpha
To verify successful deployment, visit
https://acme-registry-alpha.appspot.com/registrar in your browser (adjusting
@ -171,11 +123,12 @@ you need to configure the system and grant access to your Google account. It's
time to go to the next step, configuration.
Configuration is handled by editing code, rebuilding the project, and deploying
again. See the [configuration guide](./configuration.md) for more details. Once
you have completed basic configuration (including most critically the project ID
in your copy of `ProductionRegistryConfigExample`), you can rebuild and start
using the `nomulus` tool to create test entities in your newly deployed system.
See the [first steps tutorial](./first-steps-tutorial.md) for more information.
again. See the [configuration guide](./configuration.md) for more details.
Once you have completed basic configuration (including most critically the
project ID, client id and secret in your copy of the `nomulus-config-*.yaml`
files), you can rebuild and start using the `nomulus` tool to create test
entities in your newly deployed system. See the [first steps tutorial](./first-steps-tutorial.md)
for more information.
[app-engine-sdk]: https://cloud.google.com/appengine/docs/java/download
[java-jdk11]: https://www.oracle.com/java/technologies/javase-downloads.html