Expand list of system capabilities in README and add known issues

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131074174

README.md

@@ -8,6 +8,9 @@ code what [Jon Postel][postel] used to do on index cards.
 This is the software that [Google Registry][google-registry] uses to operate
 TLDs such as .GOOGLE, .HOW, .SOY, and .みんな.
 
+For more in-depth documentation, including install and setup instructions, see
+the Markdown documents in the `docs` directory.
+
 ### What is a Registry?
 
 When it comes to internet land, ownership flows down the following hierarchy:
@@ -75,29 +78,10 @@ consistency) and then rewinding the entities to the desired point in time.
 The Domain Registry codebase is also well tested. The core packages in the
 codebase (model, flows, rde, whois, etc.) have 95% test coverage.
 
+## Capabilities
 
-## Setup
+Domain Registry has the following capabilities, many of which are standard IETF
-
+services.
-All you have to do is [install Bazel][bazel-install] and clone this repository
-locally. Once that's done, here are some example commands to get you started:
-
-```sh
-# Run all tests
-bazel test //java{,tests}/google/registry/...
-
-# Run the registry_tool command
-bazel run //java/google/registry/tool:registry_tool -- --help
-
-# Run the Registrar Console on a local development server
-bazel run //javatests/google/registry/server -- --help
-bazel run //javatests/google/registry/server
-google-chrome http://localhost:8080/registrar
-```
-
-
-## Services
-
-Domain Registry provides the following IETF standard services.
 
 ### Extensible Provisioning Protocol (EPP)
 
@@ -105,21 +89,19 @@ Domain Registry provides the following IETF standard services.
 by registrars to register domains from the registry on behalf of registrants.
 Domain Registry implements this service as an App Engine HTTP servlet listening
 on the `/_dr/epp` path. Requests are forwarded to this path by a public-facing
-proxy listening on port 700.
+proxy listening on port 700. Poll message support is also included.
 
 To supplement EPP, Domain Registry also provides a public API for performing
 domain availability checks. This service listens on the `/check` path.
 
-#### RFCs
+* [RFC 5730: EPP](http://tools.ietf.org/html/rfc5730)
-
+* [RFC 5731: EPP Domain Mapping](http://tools.ietf.org/html/rfc5731)
-- [RFC 5730: EPP](http://tools.ietf.org/html/rfc5730)
+* [RFC 5732: EPP Host Mapping](http://tools.ietf.org/html/rfc5732)
-- [RFC 5731: EPP Domain Mapping](http://tools.ietf.org/html/rfc5731)
+* [RFC 5733: EPP Contact Mapping](http://tools.ietf.org/html/rfc5733)
-- [RFC 5732: EPP Host Mapping](http://tools.ietf.org/html/rfc5732)
+* [RFC 3915: EPP Grace Period Mapping](http://tools.ietf.org/html/rfc3915)
-- [RFC 5733: EPP Contact Mapping](http://tools.ietf.org/html/rfc5733)
+* [RFC 5734: EPP Transport over TCP](http://tools.ietf.org/html/rfc5734)
-- [RFC 3915: EPP Grace Period Mapping](http://tools.ietf.org/html/rfc3915)
+* [RFC 5910: EPP DNSSEC Mapping](http://tools.ietf.org/html/rfc5910)
-- [RFC 5734: EPP Transport over TCP](http://tools.ietf.org/html/rfc5734)
+* [Draft: EPP Launch Phase Mapping (Proposed)](http://tools.ietf.org/html/draft-tan-epp-launchphase-11)
-- [RFC 5910: EPP DNSSEC Mapping](http://tools.ietf.org/html/rfc5910)
-- [Draft: EPP Launch Phase Mapping (Proposed)](http://tools.ietf.org/html/draft-tan-epp-launchphase-11)
 
 ### Registry Data Escrow (RDE)
 
@@ -132,21 +114,17 @@ This service exists for ICANN regulatory purposes. ICANN needs to know that,
 should a registry business ever implode, that they can quickly migrate their
 TLDs to a different company so that they'll continue to operate.
 
-#### RFCs
+* [Draft: Registry Data Escrow Specification](http://tools.ietf.org/html/draft-arias-noguchi-registry-data-escrow-06)
-
+* [Draft: Domain Name Registration Data (DNRD) Objects Mapping](http://tools.ietf.org/html/draft-arias-noguchi-dnrd-objects-mapping-05)
-- [Draft: Registry Data Escrow Specification](http://tools.ietf.org/html/draft-arias-noguchi-registry-data-escrow-06)
+* [Draft: ICANN Registry Interfaces](http://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-05)
-- [Draft: Domain Name Registration Data (DNRD) Objects Mapping](http://tools.ietf.org/html/draft-arias-noguchi-dnrd-objects-mapping-05)
-- [Draft: ICANN Registry Interfaces](http://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-05)
 
 ### Trademark Clearing House (TMCH)
 
 Domain Registry integrates with ICANN and IBM's MarksDB in order to protect
 trademark holders, when new TLDs are being launched.
 
-#### RFCs
+* [Draft: TMCH Functional Spec](http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08)
-
+* [Draft: Mark and Signed Mark Objects Mapping](https://tools.ietf.org/html/draft-lozano-tmch-smd-02)
-- [Draft: TMCH Functional Spec](http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08)
-- [Draft: Mark and Signed Mark Objects Mapping](https://tools.ietf.org/html/draft-lozano-tmch-smd-02)
 
 ### WHOIS
 
@@ -156,10 +134,8 @@ internal HTTP endpoint running on `/_dr/whois`. A separate proxy running on port
 43 forwards requests to that path. Domain Registry also implements a public HTTP
 endpoint that listens on the `/whois` path.
 
-#### RFCs
+* [RFC 3912: WHOIS Protocol Specification](https://tools.ietf.org/html/rfc3912)
-
+* [RFC 7485: Inventory and Analysis of Registration Objects](http://tools.ietf.org/html/rfc7485)
-- [RFC 3912: WHOIS Protocol Specification](https://tools.ietf.org/html/rfc3912)
-- [RFC 7485: Inventory and Analysis of Registration Objects](http://tools.ietf.org/html/rfc7485)
 
 ### Registration Data Access Protocol (RDAP)
 
@@ -167,19 +143,127 @@ RDAP is the new standard for WHOIS. It provides much richer functionality, such
 as the ability to perform wildcard searches. Domain Registry makes this HTTP
 service available under the `/rdap/...` path.
 
-#### RFCs
+* [RFC 7480: RDAP HTTP Usage](http://tools.ietf.org/html/rfc7480)
+* [RFC 7481: RDAP Security Services](http://tools.ietf.org/html/rfc7481)
+* [RFC 7482: RDAP Query Format](http://tools.ietf.org/html/rfc7482)
+* [RFC 7483: RDAP JSON Responses](http://tools.ietf.org/html/rfc7483)
+* [RFC 7484: RDAP Finding the Authoritative Registration Data](http://tools.ietf.org/html/rfc7484)
 
-- [RFC 7480: RDAP HTTP Usage](http://tools.ietf.org/html/rfc7480)
+### Backups
-- [RFC 7481: RDAP Security Services](http://tools.ietf.org/html/rfc7481)
-- [RFC 7482: RDAP Query Format](http://tools.ietf.org/html/rfc7482)
-- [RFC 7483: RDAP JSON Responses](http://tools.ietf.org/html/rfc7483)
-- [RFC 7484: RDAP Finding the Authoritative Registration Data](http://tools.ietf.org/html/rfc7484)
 
+The registry provides a system for generating and restoring from backups with
+strong point-in-time consistency.  Datastore backups are written out once daily
+to Cloud Storage using the built-in Datastore snapshot export functionality.
+Separately, entities called commit logs are continuously exported to track
+changes that occur in between the regularly scheduled backups.
 
+A restore involves wiping out all entities in Datastore, importing the most
+recent complete daily backup snapshot, then replaying all of the commit logs
+since that snapshot.  This yields a system state that is guaranteed
+transactionally consistent.
+
+### Billing
+
+The registry performs a regular daily export of `BillingEvent` entities from
+Cloud Datastore, where they are stored and updated by the running system, to
+[BigQuery][bigquery], where they can be analyzed using SQL scripts to generate
+monthly invoices per registrar.
+
+### High availablity with horizontal scaling
+
+Because the registry runs on the Google Cloud Platform stack, it benefits from
+high availability, automatic fail-over, and horizontal auto-scaling of compute
+and database resources.  This makes it quite flexible for running TLDs of any
+size.
+
+### Automated tests
+
+The registry codebase includes ~400 test classes with ~4,000 total unit and
+integration tests.  This limits regressions, ensures correct system
+functionality, and allows for easy continued future development and refactoring.
+
+### DNS
+
+An interface for DNS operations is provided, along with a sample implementation
+that uses the [Google Cloud DNS](https://cloud.google.com/dns/) API.  A bulk
+export tool is also provided to export a zone file for an entire TLD in BIND
+format.
+
+* [RFC 1034: Domain Names - Concepts and Facilities](https://www.ietf.org/rfc/rfc1034.txt)
+* [RFC 1035: Domain Names - Implementation and Specification](https://www.ietf.org/rfc/rfc1034.txt)
+
+### Exports
+
+The registry uses background batch processes to periodically export information
+from the running system, including billing information, all EPP entities,
+backups, lists of all registered domain names, registrar contact emails,
+ICANN-mandated reports, database snapshots, and reserved terms.
+
+### Metrics and reporting
+
+The registry records metrics and regularly exports them to BigQuery so that
+analyses can be run on them using full SQL queries.  Metrics include which EPP
+commands were run and when and by whom, information on failed commands, activity
+per registrar, and length of each request.
+
+[BigQuery][bigquery] reporting scripts are provided to generate the required
+per-TLD monthly
+[registry reports](https://www.icann.org/resources/pages/registry-reports) for
+ICANN.
+
+### Registrar console
+
+The registry includes a web-based registrar console that registrars can access
+in a browser.  It provides the ability for registrars to view their billing
+invoices in Google Drive, contact the registry provider, and modify WHOIS,
+security (including SSL certificates), and registrar contact settings.  Main
+registry commands such as creating domains, hosts, and contacts must go through
+EPP and are not provided in the console.
+
+### Admin tooling
+
+The registry comes with a fully featured `registry_tool` command-line tool (see
+`docs/` for full documentation) that allows developers and support personnel of
+the registry to run a full range of commands, including creating new registrars,
+running arbitrary EPP commands, inspecting the state of important things in the
+system, and creating new TLDs.
+
+### Plug-and-play pricing engines
+
+The registry has the ability to configure per-TLD pricing engines to
+programmatically determine the price of domain names on the fly.  An
+implementation is provided that uses the contents of a static list of prices
+(this being by far the most common type of premium pricing used for TLDs).
+
+## Known issues
+
+There are a few things that the registry cannot currently do, and a few things
+that are out of scope that it will never do.
+
+* You will need a DNS system in order to run a fully-fledged registry.  If you
+  are planning on using anything other than Google Cloud DNS you will need to
+  provide an implementation.
+* You will need an invoicing system to convert the internal registry billing
+  events into registrar invoices using whatever accounts receivable setup you
+  already have.  A partial implementation is provided that generates generic CSV
+  invoices (see `MakeBillingTablesCommand`), but you will need to integrate it
+  with your payments system.
+* You will likely need monitoring to continuously monitor the status of the
+  system.  Any of a large variety of tools can be used for this, or you can
+  write your own.
+* You will need a proxy to forward traffic on EPP and WHOIS ports to the HTTPS
+  endpoint on App Engine, as App Engine only allows incoming traffic on
+  HTTP/HTTPS ports.  Similarly, App Engine does not yet support IPv6, so your
+  proxy would have to support that as well if you need IPv6 support.  Future
+  versions of [App Engine Flexible][flex] should provide these out of the box,
+  but they aren't ready yet.
+
+[bigquery]: https://cloud.google.com/bigquery/
 [datastore]: https://cloud.google.com/datastore/docs/concepts/overview
 [gae]: https://cloud.google.com/appengine/docs/about-the-standard-environment
 [bazel-install]: http://bazel.io/docs/install.html
 [epp]: https://en.wikipedia.org/wiki/Extensible_Provisioning_Protocol
+[flex]: https://cloud.google.com/appengine/docs/flexible/
 [google-registry]: https://www.registry.google/
 [gtld]: https://en.wikipedia.org/wiki/Generic_top-level_domain
 [icann]: https://en.wikipedia.org/wiki/ICANN

docs/install.md

@@ -64,6 +64,24 @@ computer too badly.
       --local_resources=1000,3,1.0
     Executed 360 out of 360 tests: 360 tests pass.
 
+## Running a development instance locally
+
+`RegistryTestServer` is a lightweight test server for the registry that is
+suitable for running locally for development.  It uses local versions of all
+Google Cloud Platform dependencies, when available.  Correspondingly, its
+functionality is limited compared to a Domain Registry instance running on an
+actual App Engine instance.  To see its command-line parameters, run:
+
+    $ bazel run //javatests/google/registry/server -- --help
+
+Then to fire up an instance of the server, run:
+
+    $ bazel run //javatests/google/registry/server {your params}
+
+Once it is running, you can interact with it via normal `registry_tool`
+commands, or view the registrar console in a web browser by navigating to
+http://localhost:8080/registrar .
+
 ## Deploying the code
 
 You are going to need to configure a variety of things before a working

docs/registry-tool.md

@@ -9,6 +9,15 @@ if the most recent export failed.  Its code lives inside the tools package
 (`java/google/registry/tools`), and is compiled by building the `registry_tool`
 target in the Bazel BUILD file in that package.
 
+To build the tool and display its command-line help, execute this command:
+
+    $ bazel run //java/google/registry/tool:registry_tool -- --help
+
+For future invocations you should alias the compiled binary in the
+`bazel-genfiles/java/google/registry` directory or add it to your path so that
+you can run it more easily.  The rest of this guide assumes that it has been
+aliased to `registry_tool`.
+
 The registry tool is always called with a specific environment to run in using
 the -e parameter.  This looks like: