zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-06-05 03:57:29 +02:00
Code Issues Projects Releases Packages Wiki Activity

Resolved merge conflict

Browse source
This commit is contained in:
Michael Muller 2018-10-31 14:41:05 -04:00
commit 02c75ee37e
237 changed files with 4148 additions and 3993 deletions
Download patch file Download diff file Expand all files Collapse all files

11
.gitignore vendored
View file

@ -76,3 +76,14 @@ autogenerated/
# Python Ignores
*.pyc
######################################################################
# Gradle Ignores
/gradle/.gradle
/gradle/gradle
/gradle/gradlew
/gradle/gradlew.bat
/gradle/**/WEB-INF
/gradle/**/build

6
docs/install.md
View file

@ -7,8 +7,8 @@ This document covers the steps necessary to download, build, and deploy Nomulus.
You will need the following programs installed on your local machine:
* A recent version of the [Java 8 JDK][java-jdk8].
* [Bazel build system](http://bazel.io/) (version [0.17.1][bazel-version]
works as of 2018-09-14).
* [Bazel build system](http://bazel.io/) (version [0.17.2][bazel-version]
works as of 2018-10-03).
* [Google App Engine SDK for Java][app-engine-sdk], and configure aliases to
to the `gcloud` and `appcfg.sh` utilities (you'll use them a lot).
* [Git](https://git-scm.com/) version control system.
@ -181,4 +181,4 @@ See the [first steps tutorial](./first-steps-tutorial.md) for more information.
[app-engine-sdk]: https://cloud.google.com/appengine/docs/java/download
[java-jdk8]: http://www.oracle.com/technetwork/java/javase/downloads
[bazel-version]: https://github.com/bazelbuild/bazel/releases/download/0.17.1/bazel-0.17.1-installer-linux-x86_64.sh
[bazel-version]: https://github.com/bazelbuild/bazel/releases/download/0.17.2/bazel-0.17.2-installer-linux-x86_64.sh

63
gradle/README.md Normal file
View file

@ -0,0 +1,63 @@
This folder contains experimental Gradle scripts as an alternative to Bazel for
the open-source Nomulus project. These are work-in-progress and are expected to
evolve in the near future.
All testing is done with Gradle v4.10.2.
## Current status
Currently there are two sub-projects, third_party, which contains the
back-ported JUnit 4.13 code; and core, which contains all Nomulus source code.
Gradle can be used to compile and run all Java tests.
Gradle is configured to use the directory containing this file as root, but use
the existing Nomulus source tree.
Dependencies are mostly the same as in Bazel, with a few exceptions:
* org.slf4j:slf4j-simple is added to provide a logging implementation in
tests. Bazel does not need this.
* com.googlecode.java-diff-utils:diffutils is not included. Bazel needs it for
Truth's equality check, but Gradle works fine without it.
* jaxb 2.2.11 is used instead of 2.3 in Bazel, since the latter breaks the
ant.xjc task. The problem is reportedly fixed in jaxb 2.4.
* The other dependencies are copied from Nomulus' repository.bzl config.
* We still need to verify if there are unused dependencies.
* Many dependencies are behind their latest versions.
### Notable Issues
Only single-threaded test execution is allowed, due to race condition over
global resources, such as the local Datastore instance, or updates to the System
properties. This is a new problem with Gradle, which does not provide as much
test isolation as Bazel. We are exploring solutions to this problem.
Test suites (RdeTestSuite and TmchTestSuite) are ignored to avoid duplicate
execution of tests. Neither suite performs any shared test setup routine, so it
is easier to exclude the suite classes than individual test classes.
Since Gradle does not support hierarchical build files, all file sets (e.g.,
resources) must be declared at the top, in root project config or the
sub-project configs.
## Initial Setup
Install Gradle on your local host, then run the following commands from this
directory:
```shell
# One-time command to add gradle wrapper:
gradle wrapper
# Start the build:
./gradlew build
```
From now on, use './gradlew build' or './gradlew test' to build and test your
changes.
To upgrade to a new Gradle version for this project, use:
```shell
gradle wrapper --gradle-version version-number
```

29
gradle/build.gradle Normal file
View file

@ -0,0 +1,29 @@
buildscript {
repositories {
jcenter()
mavenCentral()
maven {
url 'https://plugins.gradle.org/m2/'
}
}
}
allprojects {
repositories {
jcenter()
mavenCentral()
flatDir {
// The objectify jar that comes with Nomulus.
dirs "${rootDir}/../third_party/objectify/v4_1"
}
}
// Single version across all projects for now.
version = '1.0'
// Java plugin:
apply plugin: 'java'
}

366
gradle/core/build.gradle Normal file
View file

@ -0,0 +1,366 @@
plugins {
id 'java-library'
}
def javaDir = "${rootDir}/../java"
def javatestsDir = "${rootDir}/../javatests"
def generatedDir = "${project.buildDir}/generated-sources"
sourceSets {
main {
java {
srcDirs = [
"${javaDir}",
"${generatedDir}"
]
}
resources {
srcDirs = [
"${javaDir}"
]
exclude '**/*.java', '**/*.xjb'
}
}
test {
java {
srcDirs = [
"${javatestsDir}",
"${generatedDir}"
]
}
resources {
srcDirs = [
"${javatestsDir}"
]
exclude '**/*.java', '**/*.xsd', '**/*.xjb'
}
}
}
configurations {
css
jaxb
soy
}
// Relevant canned dependency labels:
// - implementation: Dependencies to be included in release distribution.
// - compileOnly: Dependencies used at compile time only for production code. They will not be
// included in release.
// - testImplementation: Dependencies needed for testing only.
dependencies {
implementation 'com.beust:jcommander:1.48'
implementation 'com.fasterxml.jackson.core:jackson-core:2.8.5'
implementation 'com.fasterxml.jackson.core:jackson-annotations:2.8.0'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.8.5'
implementation 'com.google.api-client:google-api-client:1.22.0'
implementation 'com.google.api-client:google-api-client-appengine:1.22.0'
implementation 'com.google.api-client:google-api-client-jackson2:1.20.0'
implementation 'com.google.monitoring-client:metrics:1.0.4'
implementation 'com.google.monitoring-client:stackdriver:1.0.4'
implementation 'com.google.api-client:google-api-client-java6:1.20.0'
implementation 'com.google.api-client:google-api-client-servlet:1.22.0'
implementation 'com.google.apis:google-api-services-admin-directory:directory_v1-rev72-1.22.0'
implementation 'com.google.apis:google-api-services-bigquery:v2-rev325-1.22.0'
implementation 'com.google.apis:google-api-services-clouddebugger:v2-rev8-1.22.0'
implementation 'com.google.apis:google-api-services-cloudkms:v1-rev12-1.22.0'
implementation 'com.google.apis:google-api-services-cloudresourcemanager:v1-rev6-1.22.0'
implementation 'com.google.apis:google-api-services-dataflow:v1b3-rev196-1.22.0'
implementation 'com.google.apis:google-api-services-dns:v2beta1-rev6-1.22.0'
implementation 'com.google.apis:google-api-services-drive:v2-rev160-1.19.1'
implementation 'com.google.apis:google-api-services-groupssettings:v1-rev60-1.22.0'
implementation 'com.google.apis:google-api-services-monitoring:v3-rev11-1.22.0'
implementation 'com.google.apis:google-api-services-sheets:v4-rev483-1.22.0'
implementation 'com.google.apis:google-api-services-storage:v1-rev86-1.22.0'
// TODO(b/71631624): change appengine:appengine-api-1.0-sdk to testCompileOnly after
// BillingEmailUtilsTest.java is fixed.
implementation 'com.google.appengine:appengine-api-1.0-sdk:1.9.48'
implementation 'com.google.appengine:appengine-api-labs:1.9.48'
implementation 'com.google.appengine:appengine-api-stubs:1.9.48'
implementation 'com.google.appengine.tools:appengine-gcs-client:0.6'
implementation 'com.google.appengine.tools:appengine-mapreduce:0.8.5'
implementation 'com.google.appengine.tools:appengine-pipeline:0.2.13'
implementation 'com.google.appengine:appengine-tools-sdk:1.9.48'
implementation 'com.google.auth:google-auth-library-credentials:0.7.1'
implementation 'com.google.auth:google-auth-library-oauth2-http:0.7.1'
implementation 'com.google.auto:auto-common:0.8'
implementation 'com.google.auto.factory:auto-factory:1.0-beta3'
implementation 'com.google.auto.value:auto-value-annotations:1.6.2'
implementation 'com.google.cloud.bigdataoss:gcsio:1.4.5'
implementation 'com.google.cloud.bigdataoss:util:1.4.5'
implementation 'com.google.code.findbugs:jsr305:3.0.2'
implementation 'com.google.dagger:dagger:2.15'
implementation 'com.google.dagger:dagger-producers:2.15'
implementation 'com.google.errorprone:error_prone_annotations:2.1.3'
implementation 'com.google.errorprone:javac-shaded:9-dev-r4023-3'
implementation 'com.google.flogger:flogger:0.1'
implementation 'com.google.flogger:flogger-system-backend:0.1'
implementation 'com.google.gdata:core:1.47.1'
implementation 'com.google.googlejavaformat:google-java-format:1.4'
implementation 'com.google.guava:guava:25.1-jre'
implementation 'com.google.http-client:google-http-client:1.22.0'
implementation 'com.google.http-client:google-http-client-appengine:1.22.0'
implementation 'com.google.http-client:google-http-client-jackson2:1.22.0'
implementation 'com.google.oauth-client:google-oauth-client:1.22.0'
implementation 'com.google.oauth-client:google-oauth-client-appengine:1.22.0'
implementation 'com.google.oauth-client:google-oauth-client-java6:1.22.0'
implementation 'com.google.oauth-client:google-oauth-client-jetty:1.22.0'
implementation 'com.google.oauth-client:google-oauth-client-servlet:1.22.0'
implementation 'com.google.protobuf:protobuf-java:2.6.0'
implementation 'com.google.re2j:re2j:1.1'
implementation 'com.google.template:soy:2018-03-14'
implementation 'com.googlecode.charts4j:charts4j:1.3'
implementation 'com.googlecode.json-simple:json-simple:1.1.1'
implementation 'com.ibm.icu:icu4j:57.1'
implementation 'com.jcraft:jsch:0.1.53'
implementation 'com.jcraft:jzlib:1.1.3'
implementation 'com.squareup:javapoet:1.8.0'
implementation 'com.squareup:javawriter:2.5.1'
implementation 'com.sun.activation:javax.activation:1.2.0'
implementation 'com.thoughtworks.paranamer:paranamer:2.7'
implementation 'commons-codec:commons-codec:1.6'
implementation 'commons-logging:commons-logging:1.1.1'
implementation 'dnsjava:dnsjava:2.1.7'
implementation 'io.netty:netty-buffer:4.1.28.Final'
implementation 'io.netty:netty-codec:4.1.28.Final'
implementation 'io.netty:netty-codec-http:4.1.28.Final'
implementation 'io.netty:netty-common:4.1.28.Final'
implementation 'io.netty:netty-handler:4.1.28.Final'
implementation 'io.netty:netty-resolver:4.1.28.Final'
implementation 'io.netty:netty-tcnative:2.0.12.Final'
implementation 'io.netty:netty-tcnative-boringssl-static:2.0.12.Final'
implementation 'io.netty:netty-transport:4.1.28.Final'
implementation 'it.unimi.dsi:fastutil:6.5.16'
implementation 'javax.annotation:jsr250-api:1.0'
implementation 'javax.inject:javax.inject:1'
implementation 'javax.mail:mail:1.4'
implementation 'javax.servlet:servlet-api:2.5'
implementation 'javax.xml.bind:jaxb-api:2.3.0'
implementation 'javax.xml.soap:javax.xml.soap-api:1.4.0'
implementation 'jline:jline:1.0'
implementation 'joda-time:joda-time:2.3'
implementation 'org.apache.avro:avro:1.8.2'
implementation 'org.apache.beam:beam-runners-direct-java:2.2.0'
implementation 'org.apache.beam:beam-runners-google-cloud-dataflow-java:2.1.0'
implementation 'org.apache.beam:beam-sdks-common-runner-api:2.1.0'
implementation 'org.apache.beam:beam-sdks-java-core:2.2.0'
implementation 'org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:2.1.0'
implementation 'org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.2.0'
implementation 'org.apache.commons:commons-compress:1.8.1'
implementation 'org.apache.ftpserver:ftpserver-core:1.0.6'
implementation 'org.apache.httpcomponents:httpclient:4.5.2'
implementation 'org.apache.httpcomponents:httpcore:4.4.4'
implementation 'org.apache.mina:mina-core:2.0.4'
implementation 'org.apache.sshd:sshd-core:2.0.0'
implementation 'org.apache.sshd:sshd-scp:2.0.0'
implementation 'org.apache.sshd:sshd-sftp:2.0.0'
implementation 'org.apache.tomcat:servlet-api:6.0.45'
implementation 'org.apache.tomcat:tomcat-annotations-api:8.0.5'
implementation 'org.bouncycastle:bcpg-jdk15on:1.52'
implementation 'org.bouncycastle:bcpkix-jdk15on:1.52'
implementation 'org.bouncycastle:bcprov-jdk15on:1.52'
implementation 'org.codehaus.jackson:jackson-core-asl:1.9.13'
implementation 'org.codehaus.jackson:jackson-mapper-asl:1.9.13'
implementation 'org.joda:joda-money:0.10.0'
implementation 'org.json:json:20160810'
implementation 'org.khronos:opengl-api:gl1.1-android-2.1_r1'
implementation 'org.mortbay.jetty:jetty:6.1.26'
implementation 'org.mortbay.jetty:servlet-api:2.5-20081211'
implementation 'org.mortbay.jetty:jetty-util:6.1.26'
implementation 'org.slf4j:slf4j-api:1.7.16'
implementation 'org.tukaani:xz:1.5'
implementation 'org.xerial.snappy:snappy-java:1.1.4-M3'
implementation 'org.yaml:snakeyaml:1.17'
implementation 'xerces:xmlParserAPIs:2.6.2'
implementation 'xpp3:xpp3:1.1.4c'
// Custom-built objectify jar at commit ecd5165, included in Nomulus release.
implementation name: 'objectify-4.1.3'
compileOnly 'com.google.appengine:appengine-remote-api:1.9.48' // Also testImplementation
compileOnly 'com.google.auto.service:auto-service:1.0-rc4'
compileOnly 'org.osgi:org.osgi.core:4.3.0'
annotationProcessor 'com.google.auto.value:auto-value:1.6.2'
testAnnotationProcessor 'com.google.auto.value:auto-value:1.6.2'
annotationProcessor 'com.google.dagger:dagger-compiler:2.15'
testAnnotationProcessor 'com.google.dagger:dagger-compiler:2.15'
testImplementation 'com.google.appengine:appengine-remote-api:1.9.48' // Also compileOnly
testImplementation 'com.google.appengine:appengine-testing:1.9.58'
testImplementation 'com.google.guava:guava-testlib:25.0-jre'
testImplementation 'com.google.monitoring-client:contrib:1.0.4'
testImplementation 'com.google.truth:truth:0.42'
testImplementation 'com.google.truth.extensions:truth-java8-extension:0.39'
testImplementation 'org.hamcrest:hamcrest-all:1.3'
testImplementation 'org.hamcrest:hamcrest-core:1.3'
testImplementation 'org.hamcrest:hamcrest-library:1.3'
testImplementation 'junit:junit:4.12'
testImplementation 'org.mockito:mockito-all:1.9.5'
testImplementation 'org.slf4j:slf4j-simple:1.7.16' // Not needed by Bazel
testImplementation project(':third_party')
// Dependencies needed for jaxb compilation.
// Use jaxb 2.2.11 because 2.3 is known to break the Ant task we use.
// TODO: upgrade jaxb versions to 2.4.0, already in beta by Sept 2018
jaxb 'javax.xml.bind:jaxb-api:2.2.11'
jaxb 'com.sun.activation:javax.activation:1.2.0'
jaxb 'com.sun.xml.bind:jaxb-xjc:2.2.11'
jaxb 'com.sun.xml.bind:jaxb-impl:2.2.11'
jaxb 'com.sun.xml.bind:jaxb-osgi:2.2.11'
// Dependency needed for soy to java compilation.
soy 'com.google.template:soy:2018-03-14'
// Dependencies needed for compiling stylesheets to javascript
css 'com.google.closure-stylesheets:closure-stylesheets:1.5.0'
css 'args4j:args4j:2.0.26'
}
task jaxbToJava() {
doLast {
file(generatedDir).mkdirs()
// Temp dir to hold schema and bindings files. Files must be in the same directory because
// the bindings (.xjb) file does not declare relative paths to schema (.xsd) files.
def xjcTempSourceDir = file("${temporaryDir}/xjc")
xjcTempSourceDir.mkdirs()
ant.copy(
todir: "${xjcTempSourceDir}",
overwrite: true) {
fileSet(
dir: "${javaDir}/google/registry/xml/xsd",
includes: '**.xsd')
}
ant.copy(
todir: "${xjcTempSourceDir}",
overwrite: true,
file:
"${javaDir}/google/registry/xjc/bindings.xjb")
ant.taskdef(
name: 'xjc',
classname: 'com.sun.tools.xjc.XJCTask',
classpath: configurations.jaxb.asPath)
ant.xjc(
destdir: "${generatedDir}",
binding: "${xjcTempSourceDir}/bindings.xjb",
removeOldOutput: 'yes', extension: 'true') {
project.fileTree(
dir: new File("$xjcTempSourceDir"),
include: ['**/*.xsd'])
.addToAntBuilder(ant, 'schema', FileCollection.AntType.FileSet)
// -npa: do not generate package-info.java files. They will be generated below.
arg(line: '-npa -quiet -extension')
}
exec {
workingDir "${generatedDir}"
executable "${javaDir}/google/registry/xjc/make_pkginfo.sh"
args "${javaDir}/google/registry/xjc/package-info.java.in",
"${generatedDir}/google/registry/xjc"
}
}
}
task soyToJava() {
ext.soyToJava = { javaPackage, outputDirectory, soyFiles ->
javaexec {
main = "com.google.template.soy.SoyParseInfoGenerator"
classpath configurations.soy
args "--javaPackage", "${javaPackage}",
"--outputDirectory", "${outputDirectory}",
"--javaClassNameSource", "filename",
"--allowExternalCalls", "true",
"--srcs", "${soyFiles.join(',')}"
}
}
doLast {
soyToJava('google.registry.tools.soy', "${generatedDir}/google/registry/tools/soy",
fileTree(dir: "${javaDir}/google/registry/tools/soy", include: ['**/*.soy']))
soyToJava('google.registry.ui.soy.registrar',
"${generatedDir}/google/registry/ui/soy/registrar",
fileTree(dir: "${javaDir}/google/registry/ui/soy/registrar", include: ['**/*.soy']))
soyToJava('google.registry.ui.soy',
"${generatedDir}/google/registry/ui/soy",
files {
file("${javaDir}/google/registry/ui/soy").listFiles()
}.filter {
it.name.endsWith(".soy")
})
}
}
task stylesheetsToJavascript {
ext.cssCompile = { outputName, debug, cssFiles ->
javaexec {
main = "com.google.common.css.compiler.commandline.ClosureCommandLineCompiler"
classpath configurations.css
def argsBuffer = [
"--output-file", "${outputName}.css",
"--output-source-map", "${outputName}.css.map",
"--input-orientation", "LTR",
"--output-orientation", "NOCHANGE",
"--output-renaming-map", "${outputName}.css.js",
"--output-renaming-map-format", "CLOSURE_COMPILED_SPLIT_HYPHENS"
]
if (debug) {
argsBuffer.addAll(["--rename", "DEBUG", "--pretty-print"])
} else {
argsBuffer.addAll(["--rename", "CLOSURE"])
}
argsBuffer.addAll(cssFiles)
args argsBuffer
}
}
doLast {
def cssSourceDir = "${javaDir}/google/registry/ui/css"
def outputDir = "${project.buildDir}/resources/main/google/registry/ui/css"
file("${outputDir}").mkdirs()
def srcFiles = [
"${cssSourceDir}/console.css", "${cssSourceDir}/contact-settings.css",
"${cssSourceDir}/contact-us.css", "${cssSourceDir}/dashboard.css",
"${cssSourceDir}/epp.css", "${cssSourceDir}/forms.css",
"${cssSourceDir}/kd_components.css", "${cssSourceDir}/registry.css",
"${cssSourceDir}/resources.css", "${cssSourceDir}/security-settings.css"
]
cssCompile("${outputDir}/registrar_bin", false, srcFiles)
cssCompile("${outputDir}/registrar_dbg", true, srcFiles)
}
}
compileJava.dependsOn jaxbToJava
compileJava.dependsOn soyToJava
// stylesheetsToJavascript must happen after processResources, which wipes the resources folder
// before copying data into it.
stylesheetsToJavascript.dependsOn processResources
classes.dependsOn stylesheetsToJavascript
test {
// Test exclusion patterns:
// - *TestCase.java are inherited by concrete test classes.
// - *TestSuite.java are excluded to avoid duplicate execution of suite members. See README
// in this directory for more information.
exclude "**/*TestCase.*", "**/*TestSuite.*"
// Use a single JVM to execute all tests. See README in this directory for more information.
maxParallelForks 1
// Use a single thread to execute all tests in a JVM. See README in this directory for more
// information.
forkEvery 1
// Uncomment to see test outputs in stdout.
//testLogging.showStandardStreams = true
}

5
gradle/settings.gradle Normal file
View file

@ -0,0 +1,5 @@
rootProject.name = 'nomulus'
include 'third_party'
include 'core'

14
gradle/third_party/build.gradle vendored Normal file
View file

@ -0,0 +1,14 @@
plugins {
id 'java-library'
}
sourceSets {
main {
java {
srcDirs = [
"${rootDir}/../third_party/junit/"
]
include '**/*.java'
}
}
}

2
java/google/registry/beam/invoicing/InvoicingPipeline.java
View file

@ -34,6 +34,7 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Filter;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.KV;
@ -141,6 +142,7 @@ public class InvoicingPipeline implements Serializable {
"Map to invoicing key",
MapElements.into(TypeDescriptor.of(InvoiceGroupingKey.class))
.via(BillingEvent::getInvoiceGroupingKey))
.apply(Filter.by((InvoiceGroupingKey key) -> key.unitPrice() != 0))
.setCoder(new InvoiceGroupingKeyCoder())
.apply("Count occurrences", Count.perElement())
.apply(

164
java/google/registry/config/RegistryConfig.java
View file

@ -24,16 +24,20 @@ import com.google.common.base.Supplier;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.net.HostAndPort;
import dagger.Module;
import dagger.Provides;
import google.registry.config.RegistryConfigSettings.AppEngine.ToolsServiceUrl;
import google.registry.util.RandomStringGenerator;
import google.registry.util.StringGenerator;
import google.registry.util.TaskQueueUtils;
import java.lang.annotation.Documented;
import java.lang.annotation.Retention;
import java.net.URI;
import java.net.URL;
import java.security.NoSuchAlgorithmException;
import java.security.ProviderException;
import java.security.SecureRandom;
import java.util.Optional;
import java.util.Random;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import javax.inject.Named;
@ -815,6 +819,7 @@ public final class RegistryConfig {
*
* @see google.registry.reporting.icann.ReportingEmailUtils
* @see google.registry.reporting.billing.BillingEmailUtils
* @see google.registry.reporting.spec11.Spec11EmailUtils
*/
@Provides
@Config("alertRecipientEmailAddress")
@ -827,13 +832,35 @@ public final class RegistryConfig {
*
* @see google.registry.reporting.icann.ReportingEmailUtils
* @see google.registry.reporting.billing.BillingEmailUtils
* @see google.registry.reporting.spec11.Spec11EmailUtils
*/
@Provides
@Config("alertSenderEmailAddress")
public static String provideAlertSenderEmailAddress(
@Config("projectId") String projectId, RegistryConfigSettings config) {
return String.format("%s@%s", projectId, config.misc.alertEmailSenderDomain);
return String.format("%s-no-reply@%s", projectId, config.misc.alertEmailSenderDomain);
}
/**
* Returns the email address to which spec 11 email should be replied.
*
* @see google.registry.reporting.spec11.Spec11EmailUtils
*/
@Provides
@Config("spec11ReplyToEmailAddress")
public static String provideSpec11ReplyToEmailAddress(RegistryConfigSettings config) {
return config.misc.spec11ReplyToEmailAddress;
}
/**
* Returns the template for the body of the spec 11 email to the registrars.
*
* @see google.registry.reporting.spec11.Spec11EmailUtils
*/
@Provides
@Config("spec11EmailBodyTemplate")
public static String provideSpec11EmailBodyTemplate(RegistryConfigSettings config) {
return config.registryPolicy.spec11EmailBodyTemplate;
}
/**
@ -1021,6 +1048,12 @@ public final class RegistryConfig {
return config.registryPolicy.greetingServerId;
}
@Provides
@Config("activeKeyring")
public static String provideKeyring(RegistryConfigSettings config) {
return config.keyring.activeKeyring;
}
/**
* The name to use for the Cloud KMS KeyRing containing encryption keys for Nomulus secrets.
*
@ -1030,13 +1063,13 @@ public final class RegistryConfig {
@Provides
@Config("cloudKmsKeyRing")
public static String provideCloudKmsKeyRing(RegistryConfigSettings config) {
return config.kms.keyringName;
return config.keyring.kms.keyringName;
}
@Provides
@Config("cloudKmsProjectId")
public static String provideCloudKmsProjectId(RegistryConfigSettings config) {
return config.kms.projectId;
return config.keyring.kms.projectId;
}
@Provides
@ -1061,7 +1094,7 @@ public final class RegistryConfig {
@Provides
@Config("premiumTermsExportDisclaimer")
public static String providePremiumTermsExportDisclaimer(RegistryConfigSettings config) {
return formatComments(config.registryPolicy.reservedTermsExportDisclaimer);
return formatComments(config.registryPolicy.premiumTermsExportDisclaimer);
}
/**
@ -1186,6 +1219,12 @@ public final class RegistryConfig {
return config.registryTool.clientSecretFilename;
}
@Provides
@Config("rdapTos")
public static ImmutableList<String> provideRdapTos(RegistryConfigSettings config) {
return ImmutableList.copyOf(Splitter.on('\n').split(config.registryPolicy.rdapTos));
}
/**
* Returns the help text to be used by RDAP.
*
@ -1195,7 +1234,8 @@ public final class RegistryConfig {
@Singleton
@Provides
@Config("rdapHelpMap")
public static ImmutableMap<String, RdapNoticeDescriptor> provideRdapHelpMap() {
public static ImmutableMap<String, RdapNoticeDescriptor> provideRdapHelpMap(
@Config("rdapTos") ImmutableList<String> rdapTos) {
return new ImmutableMap.Builder<String, RdapNoticeDescriptor>()
.put("/", RdapNoticeDescriptor.builder()
.setTitle("RDAP Help")
@ -1216,37 +1256,51 @@ public final class RegistryConfig {
.build())
.put("/tos", RdapNoticeDescriptor.builder()
.setTitle("RDAP Terms of Service")
.setDescription(ImmutableList.of(
"By querying our Domain Database as part of the RDAP pilot program (RDAP Domain"
+ "Database), you are agreeing to comply with these terms, so please read"
+ " them carefully.",
"Any information provided is 'as is' without any guarantee of accuracy.",
"Please do not misuse the RDAP Domain Database. It is intended solely for"
+ " query-based access on an experimental basis and should not be used for or"
+ " relied upon for any other purpose.",
"Don't use the RDAP Domain Database to allow, enable, or otherwise support the"
+ " transmission of mass unsolicited, commercial advertising or"
+ " solicitations.",
"Don't access our RDAP Domain Database through the use of high volume, automated"
+ " electronic processes that send queries or data to the systems of any"
+ " ICANN-accredited registrar.",
"You may only use the information contained in the RDAP Domain Database for"
+ " lawful purposes.",
"Do not compile, repackage, disseminate, or otherwise use the information"
+ " contained in the RDAP Domain Database in its entirety, or in any"
+ " substantial portion, without our prior written permission.",
"We may retain certain details about queries to our RDAP Domain Database for the"
+ " purposes of detecting and preventing misuse.",
"We reserve the right to restrict or deny your access to the RDAP Domain Database"
+ " if we suspect that you have failed to comply with these terms.",
"We reserve the right to modify or discontinue our participation in the RDAP"
+ " pilot program and suspend or terminate access to the RDAP Domain Database"
+ " at any time and for any reason in our sole discretion.",
"We reserve the right to modify this agreement at any time."))
.setDescription(rdapTos)
.setLinkValueSuffix("help/tos")
.build())
.build();
}
/**
* Returns a singleton insecure random number generator that is fast.
*
* <p>This binding is intentionally qualified so that any requester must explicitly acknowledge
* that using an insecure random number generator is fine for its use case.
*/
@Singleton
@Provides
@Config("insecureRandom")
public static Random provideInsecureRandom() {
return new Random();
}
/** Returns a singleton secure random number generator this is slow. */
@Singleton
@Provides
public static SecureRandom provideSecureRandom() {
try {
return SecureRandom.getInstance("NativePRNG");
} catch (NoSuchAlgorithmException e) {
throw new ProviderException(e);
}
}
/** Returns a singleton random string generator using Base58 encoding. */
@Singleton
@Provides
@Config("base58StringGenerator")
public static StringGenerator provideBase58StringGenerator(SecureRandom secureRandom) {
return new RandomStringGenerator(StringGenerator.Alphabets.BASE_58, secureRandom);
}
/** Returns a singleton random string generator using Base58 encoding. */
@Singleton
@Provides
@Config("base64StringGenerator")
public static StringGenerator provideBase64StringGenerator(SecureRandom secureRandom) {
return new RandomStringGenerator(StringGenerator.Alphabets.BASE_64, secureRandom);
}
}
/**
@ -1295,14 +1349,44 @@ public final class RegistryConfig {
return Duration.standardDays(30);
}
public static boolean areServersLocal() {
return CONFIG_SETTINGS.get().appEngine.isLocal;
}
/**
* Returns the address of the Nomulus app HTTP server.
* Returns the address of the Nomulus app default HTTP server.
*
* <p>This is used by the {@code nomulus} tool to connect to the App Engine remote API.
*/
public static HostAndPort getServer() {
ToolsServiceUrl url = CONFIG_SETTINGS.get().appEngine.toolsServiceUrl;
return HostAndPort.fromParts(url.hostName, url.port);
public static URL getDefaultServer() {
return makeUrl(CONFIG_SETTINGS.get().appEngine.defaultServiceUrl);
}
/**
* Returns the address of the Nomulus app backend HTTP server.
*
* <p>This is used by the {@code nomulus} tool to connect to the App Engine remote API.
*/
public static URL getBackendServer() {
return makeUrl(CONFIG_SETTINGS.get().appEngine.backendServiceUrl);
}
/**
* Returns the address of the Nomulus app tools HTTP server.
*
* <p>This is used by the {@code nomulus} tool to connect to the App Engine remote API.
*/
public static URL getToolsServer() {
return makeUrl(CONFIG_SETTINGS.get().appEngine.toolsServiceUrl);
}
/**
* Returns the address of the Nomulus app pubapi HTTP server.
*
* <p>This is used by the {@code nomulus} tool to connect to the App Engine remote API.
*/
public static URL getPubapiServer() {
return makeUrl(CONFIG_SETTINGS.get().appEngine.pubapiServiceUrl);
}
/** Returns the amount of time a singleton should be cached, before expiring. */
@ -1410,7 +1494,7 @@ public final class RegistryConfig {
* change the contents of the YAML config files.
*/
@VisibleForTesting
static final Supplier<RegistryConfigSettings> CONFIG_SETTINGS =
public static final Supplier<RegistryConfigSettings> CONFIG_SETTINGS =
memoize(YamlUtils::getConfigSettings);
private static String formatComments(String text) {

36
java/google/registry/config/RegistryConfigSettings.java
View file

@ -34,19 +34,17 @@ public class RegistryConfigSettings {
public Monitoring monitoring;
public Misc misc;
public Beam beam;
public Kms kms;
public Keyring keyring;
public RegistryTool registryTool;
/** Configuration options that apply to the entire App Engine project. */
public static class AppEngine {
public String projectId;
public ToolsServiceUrl toolsServiceUrl;
/** Configuration options for the tools service URL. */
public static class ToolsServiceUrl {
public String hostName;
public int port;
}
public boolean isLocal;
public String defaultServiceUrl;
public String backendServiceUrl;
public String toolsServiceUrl;
public String pubapiServiceUrl;
}
/** Configuration options for OAuth settings for authenticating users. */
@ -81,7 +79,6 @@ public class RegistryConfigSettings {
public String greetingServerId;
public List<String> registrarChangesNotificationEmailAddresses;
public String defaultRegistrarWhoisServer;
public String defaultRegistrarReferralUrl;
public String tmchCaMode;
public String tmchCrlUrl;
public String tmchMarksDbUrl;
@ -90,6 +87,8 @@ public class RegistryConfigSettings {
public String premiumTermsExportDisclaimer;
public String reservedTermsExportDisclaimer;
public String whoisDisclaimer;
public String rdapTos;
public String spec11EmailBodyTemplate;
}
/** Configuration for Cloud Datastore. */
@ -99,12 +98,6 @@ public class RegistryConfigSettings {
public int baseOfyRetryMillis;
}
/** Configuration for Cloud KMS. */
public static class Kms {
public String keyringName;
public String projectId;
}
/** Configuration for Apache Beam (Cloud Dataflow). */
public static class Beam {
public String defaultJobZone;
@ -166,10 +159,23 @@ public class RegistryConfigSettings {
public static class Misc {
public String sheetExportId;
public String alertRecipientEmailAddress;
public String spec11ReplyToEmailAddress;
public String alertEmailSenderDomain;
public int asyncDeleteDelaySeconds;
}
/** Configuration for keyrings (used to store secrets outside of source). */
public static class Keyring {
public String activeKeyring;
public Kms kms;
}
/** Configuration for Cloud KMS. */
public static class Kms {
public String keyringName;
public String projectId;
}
/** Configuration options for the registry tool. */
public static class RegistryTool {
public String clientSecretFilename;

100
java/google/registry/config/files/default-config.yaml
View file

@ -9,10 +9,13 @@ appEngine:
# Globally unique App Engine project ID
projectId: registry-project-id
# Hostname and port of the tools service for the project.
toolsServiceUrl:
hostName: localhost
port: 443
# whether to use local/test credentials when connecting to the servers
isLocal: true
# URLs of the services for the project.
defaultServiceUrl: https://localhost
backendServiceUrl: https://localhost
toolsServiceUrl: https://localhost
pubapiServiceUrl: https://localhost
gSuite:
# Publicly accessible domain name of the running G Suite instance.
@ -103,6 +106,72 @@ registryPolicy:
unlawful behavior. We reserve the right to restrict or deny your access to
the WHOIS database, and may modify these terms at any time.
# RDAP Terms of Service text displayed at the /rdap/help/tos endpoint.
rdapTos: >
By querying our Domain Database as part of the RDAP pilot program (RDAP
Domain Database), you are agreeing to comply with these terms, so please
read them carefully.
Any information provided is 'as is' without any guarantee of accuracy.
Please do not misuse the RDAP Domain Database. It is intended solely for
query-based access on an experimental basis and should not be used for or
relied upon for any other purpose.
Don't use the RDAP Domain Database to allow, enable, or otherwise support
the transmission of mass unsolicited, commercial advertising or
solicitations.
Don't access our RDAP Domain Database through the use of high volume,
automated electronic processes that send queries or data to the systems
of any ICANN-accredited registrar.
You may only use the information contained in the RDAP Domain Database for
lawful purposes.
Do not compile, repackage, disseminate, or otherwise use the information
contained in the RDAP Domain Database in its entirety, or in any
substantial portion, without our prior written permission.
We may retain certain details about queries to our RDAP Domain Database
for the purposes of detecting and preventing misuse.
We reserve the right to restrict or deny your access to the RDAP Domain
Database if we suspect that you have failed to comply with these terms.
We reserve the right to modify or discontinue our participation in the
RDAP pilot program and suspend or terminate access to the RDAP Domain
Database at any time and for any reason in our sole discretion.
We reserve the right to modify this agreement at any time.
# Body of the spec 11 email sent to registrars.
# Items in braces are to be replaced.
spec11EmailBodyTemplate: |
Dear registrar partner,
The registry conducts periodic technical analyses of all domains registered
in its TLDs. As part of this analysis, the following domains that you
manage were flagged for potential security concerns:
{LIST_OF_THREATS}
Please communicate these findings to the registrant and work with the
registrant to mitigate any security issues and have the domains delisted.
Some helpful sites for getting off a blocked list include:
- Google Search Console (https://search.google.com/search-console/about)
-- includes information and tools for webmasters to learn about and
mitigate security threats and have their websites delisted
- first.org -- a registry of Computer Emergency Response Teams (CERTs)
that may be able to assist in mitigation
- stopbadware.org -- a non-profit anti-malware organization that provides
support and information for webmasters dealing with security threats
If you have any questions regarding this notice, please contact
{REPLY_TO_EMAIL}.
datastore:
# Number of commit log buckets in Datastore. Lowering this after initial
# install risks losing up to a days' worth of differential backups.
@ -266,6 +335,10 @@ misc:
# Address we send alert summary emails to.
alertRecipientEmailAddress: email@example.com
# Address to which the Spec 11 emails to registrars should be replied. This needs
# to be a deliverable email address in case the registrars want to contact us.
spec11ReplyToEmailAddress: reply-to@example.com
# Domain for the email address we send alert summary emails from.
alertEmailSenderDomain: appspotmail.com
@ -283,14 +356,19 @@ beam:
# The default zone to run Apache Beam (Cloud Dataflow) jobs in.
defaultJobZone: us-east1-c
kms:
# GCP project containing the KMS keyring. Should only be used for KMS in
# order to keep a simple locked down IAM configuration.
projectId: registry-kms-project-id
keyring:
# The name of the active keyring, either "KMS" or "Dummy".
activeKeyring: Dummy
# The name to use for the Cloud KMS KeyRing which will store encryption keys
# for Nomulus secrets.
keyringName: nomulus
# Configuration options specific to Google Cloud KMS.
kms:
# GCP project containing the KMS keyring. Should only be used for KMS in
# order to keep a simple locked down IAM configuration.
projectId: registry-kms-project-id
# The name to use for the Cloud KMS KeyRing which will store encryption keys
# for Nomulus secrets.
keyringName: nomulus
# Configuration options relevant to the "nomulus" registry tool.
registryTool:

19
java/google/registry/config/files/nomulus-config-production-sample.yaml
View file

@ -4,11 +4,14 @@
appEngine:
projectId: placeholder
# The "tools-dot-" prefix is used on the project ID in this URL in order to
# get around an issue with double-wildcard SSL certs.
toolsServiceUrl:
hostName: tools-dot-placeholder.appspot.com
port: 443
# Set to true if running against local servers (localhost)
isLocal: false
# The "<service>-dot-" prefix is used on the project ID in this URL in order
# to get around an issue with double-wildcard SSL certs.
defaultServiceUrl: https://domain-registry-placeholder.appspot.com
backendServiceUrl: https://backend-dot-domain-registry-placeholder.appspot.com
toolsServiceUrl: https://tools-dot-domain-registry-placeholder.appspot.com
pubapiServiceUrl: https://pubapi-dot-domain-registry-placeholder.appspot.com
gSuite:
domainName: placeholder
@ -61,5 +64,7 @@ cloudDns:
rootUrl: null
servicePath: null
kms:
projectId: placeholder
keyring:
activeKeyring: KMS
kms:
projectId: placeholder

11
java/google/registry/env/common/default/WEB-INF/queue.xml vendored
View file

@ -192,17 +192,6 @@
</retry-parameters>
</queue>
<!-- queue for whitebox metrics -->
<queue>
<name>bigquery-streaming-metrics</name>
<rate>500/s</rate>
<bucket-size>500</bucket-size>
<retry-parameters>
<task-retry-limit>1</task-retry-limit>
<task-age-limit>1m</task-age-limit>
</retry-parameters>
</queue>
<!-- Queue for infrequent cron tasks (i.e. hourly or less often) that should retry three times on failure. -->
<queue>
<name>retryable-cron-tasks</name>

6
java/google/registry/flows/EppController.java
View file

@ -22,7 +22,6 @@ import static google.registry.flows.FlowReporter.extractTlds;
import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Joiner;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableMap;
import com.google.common.flogger.FluentLogger;
@ -33,7 +32,6 @@ import google.registry.model.eppoutput.EppOutput;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.eppoutput.Result;
import google.registry.model.eppoutput.Result.Code;
import google.registry.monitoring.whitebox.BigQueryMetricsEnqueuer;
import google.registry.monitoring.whitebox.EppMetric;
import java.util.Optional;
import javax.inject.Inject;
@ -52,7 +50,6 @@ public final class EppController {
@Inject FlowComponent.Builder flowComponentBuilder;
@Inject EppMetric.Builder eppMetricBuilder;
@Inject EppMetrics eppMetrics;
@Inject BigQueryMetricsEnqueuer bigQueryMetricsEnqueuer;
@Inject ServerTridProvider serverTridProvider;
@Inject EppController() {}
@ -65,7 +62,6 @@ public final class EppController {
boolean isSuperuser,
byte[] inputXmlBytes) {
eppMetricBuilder.setClientId(Optional.ofNullable(sessionMetadata.getClientId()));
eppMetricBuilder.setPrivilegeLevel(isSuperuser ? "SUPERUSER" : "NORMAL");
try {
EppInput eppInput;
try {
@ -99,7 +95,6 @@ public final class EppController {
e.getResult(), Trid.create(null, serverTridProvider.createServerTrid()));
}
if (!eppInput.getTargetIds().isEmpty()) {
eppMetricBuilder.setEppTarget(Joiner.on(',').join(eppInput.getTargetIds()));
if (eppInput.isDomainResourceType()) {
eppMetricBuilder.setTlds(extractTlds(eppInput.getTargetIds()));
}
@ -122,7 +117,6 @@ public final class EppController {
} finally {
if (!isDryRun) {
EppMetric metric = eppMetricBuilder.build();
bigQueryMetricsEnqueuer.export(metric);
eppMetrics.incrementEppRequests(metric);
eppMetrics.recordProcessingTime(metric);
}

45
java/google/registry/flows/EppXmlSanitizer.java
View file

@ -14,14 +14,18 @@
package google.registry.flows;
import static com.google.common.base.Preconditions.checkState;
import com.google.common.base.CharMatcher;
import com.google.common.collect.ImmutableSet;
import com.google.common.flogger.FluentLogger;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.UnsupportedEncodingException;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Locale;
import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.xml.namespace.QName;
@ -32,6 +36,7 @@ import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.events.Characters;
import javax.xml.stream.events.StartDocument;
import javax.xml.stream.events.XMLEvent;
/**
@ -71,31 +76,47 @@ public class EppXmlSanitizer {
private static final XMLEventFactory XML_EVENT_FACTORY = XMLEventFactory.newFactory();
/**
* Returns sanitized and pretty-printed EPP XML message. For malformed XML messages,
* base64-encoded raw bytes will be returned.
* Returns sanitized EPP XML message. For malformed XML messages, base64-encoded raw bytes will be
* returned.
*
* <p>The xml version header {@code <?xml version="1.0" ?>} is added to the result if the input
* does not already contain it. Also, an empty element will be formatted as {@code <tag></tag>}
* instead of {@code <tag/>}.
* <p>The output always begins with version and encoding declarations no matter if the input
* includes them. If encoding is not declared by input, UTF-8 will be used according to XML
* standard.
*
* <p>Also, an empty element will be formatted as {@code <tag></tag>} instead of {@code <tag/>}.
*/
public static String sanitizeEppXml(byte[] inputXmlBytes) {
try {
// Keep exactly one newline at end of sanitized string.
return CharMatcher.whitespace()
.trimTrailingFrom(new String(sanitize(inputXmlBytes), StandardCharsets.UTF_8))
+ "\n";
} catch (XMLStreamException e) {
return CharMatcher.whitespace().trimTrailingFrom(sanitizeAndEncode(inputXmlBytes)) + "\n";
} catch (XMLStreamException | UnsupportedEncodingException e) {
logger.atWarning().withCause(e).log("Failed to sanitize EPP XML message.");
return Base64.getMimeEncoder().encodeToString(inputXmlBytes);
}
}
private static byte[] sanitize(byte[] inputXmlBytes) throws XMLStreamException {
private static String sanitizeAndEncode(byte[] inputXmlBytes)
throws XMLStreamException, UnsupportedEncodingException {
XMLEventReader xmlEventReader =
XML_INPUT_FACTORY.createXMLEventReader(new ByteArrayInputStream(inputXmlBytes));
if (!xmlEventReader.hasNext()) {
return "";
}
XMLEvent firstEvent = xmlEventReader.nextEvent();
checkState(firstEvent.isStartDocument(), "Missing StartDocument");
// Get input encoding for use in XMLEventWriter creation, so that sanitized XML preserves the
// encoding declaration. According to XML spec, UTF-8 is to be used unless input declares
// otherwise. Epp officially allows UTF-8 and UTF-16.
String inputEncoding =
Optional.ofNullable(((StartDocument) firstEvent).getCharacterEncodingScheme())
.orElse(StandardCharsets.UTF_8.name());
ByteArrayOutputStream outputXmlBytes = new ByteArrayOutputStream();
XMLEventWriter xmlEventWriter = XML_OUTPUT_FACTORY.createXMLEventWriter(outputXmlBytes);
XMLEventWriter xmlEventWriter =
XML_OUTPUT_FACTORY.createXMLEventWriter(outputXmlBytes, inputEncoding);
xmlEventWriter.add(firstEvent);
while (xmlEventReader.hasNext()) {
XMLEvent xmlEvent = xmlEventReader.nextEvent();
@ -118,7 +139,7 @@ public class EppXmlSanitizer {
}
}
xmlEventWriter.flush();
return outputXmlBytes.toByteArray();
return outputXmlBytes.toString(inputEncoding);
}
private static String maskSensitiveData(String original) {

2
java/google/registry/flows/FlowRunner.java
View file

@ -72,7 +72,6 @@ public class FlowRunner {
}
eppMetricBuilder.setCommandNameFromFlow(flowClass.getSimpleName());
if (!isTransactional) {
eppMetricBuilder.incrementAttempts();
EppOutput eppOutput = EppOutput.create(flowProvider.get().run());
if (flowClass.equals(LoginFlow.class)) {
// In LoginFlow, clientId isn't known until after the flow executes, so save it then.
@ -84,7 +83,6 @@ public class FlowRunner {
return ofy()
.transact(
() -> {
eppMetricBuilder.incrementAttempts();
try {
EppOutput output = EppOutput.create(flowProvider.get().run());
if (isDryRun) {

10
java/google/registry/flows/async/AsyncFlowMetrics.java
View file

@ -29,7 +29,6 @@ import com.google.monitoring.metrics.IncrementableMetric;
import com.google.monitoring.metrics.LabelDescriptor;
import com.google.monitoring.metrics.MetricRegistryImpl;
import google.registry.util.Clock;
import google.registry.util.NonFinalForTesting;
import javax.inject.Inject;
import org.joda.time.DateTime;
import org.joda.time.Duration;
@ -67,9 +66,8 @@ public class AsyncFlowMetrics {
LabelDescriptor.create("operation_type", "The type of async flow operation."),
LabelDescriptor.create("result", "The result of the async flow operation."));
@NonFinalForTesting
@VisibleForTesting
static IncrementableMetric asyncFlowOperationCounts =
static final IncrementableMetric asyncFlowOperationCounts =
MetricRegistryImpl.getDefault()
.newIncrementableMetric(
"/async_flows/operations",
@ -77,9 +75,8 @@ public class AsyncFlowMetrics {
"count",
LABEL_DESCRIPTORS);
@NonFinalForTesting
@VisibleForTesting
static EventMetric asyncFlowOperationProcessingTime =
static final EventMetric asyncFlowOperationProcessingTime =
MetricRegistryImpl.getDefault()
.newEventMetric(
"/async_flows/processing_time",
@ -88,9 +85,8 @@ public class AsyncFlowMetrics {
LABEL_DESCRIPTORS,
DEFAULT_FITTER);
@NonFinalForTesting
@VisibleForTesting
static EventMetric asyncFlowBatchSize =
static final EventMetric asyncFlowBatchSize =
MetricRegistryImpl.getDefault()
.newEventMetric(
"/async_flows/batch_size",

7
java/google/registry/flows/domain/DomainAllocateFlow.java
View file

@ -88,7 +88,7 @@ import google.registry.model.reporting.DomainTransactionRecord;
import google.registry.model.reporting.DomainTransactionRecord.TransactionReportField;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.tmch.LordnTask;
import google.registry.tmch.LordnTaskUtils;
import java.util.Optional;
import java.util.Set;
import javax.inject.Inject;
@ -385,13 +385,14 @@ public class DomainAllocateFlow implements TransactionalFlow {
dnsQueue.get().addDomainRefreshTask(newDomain.getFullyQualifiedDomainName());
}
if (allocateCreate.getSmdId() != null || allocateCreate.getNotice() != null) {
LordnTask.enqueueDomainResourceTask(newDomain);
LordnTaskUtils.enqueueDomainResourceTask(newDomain);
}
}
private ImmutableList<FeeTransformResponseExtension> createResponseExtensions(
DateTime now, Registry registry, int years) throws EppException {
FeesAndCredits feesAndCredits = pricingLogic.getCreatePrice(registry, targetId, now, years);
FeesAndCredits feesAndCredits =
pricingLogic.getCreatePrice(registry, targetId, now, years, false);
Optional<FeeCreateCommandExtension> feeCreate =
eppInput.getSingleExtension(FeeCreateCommandExtension.class);
return feeCreate.isPresent()

7
java/google/registry/flows/domain/DomainApplicationCreateFlow.java
View file

@ -211,8 +211,11 @@ public final class DomainApplicationCreateFlow implements TransactionalFlow {
checkAllowedAccessToTld(clientId, tld);
}
Registry registry = Registry.get(tld);
boolean isAnchorTenant =
isAnchorTenant(domainName, Optional.empty(), authInfo.getPw().getValue(), Optional.empty());
FeesAndCredits feesAndCredits =
pricingLogic.getCreatePrice(registry, targetId, now, command.getPeriod().getValue());
pricingLogic.getCreatePrice(
registry, targetId, now, command.getPeriod().getValue(), isAnchorTenant);
verifyUnitIsYears(command.getPeriod());
int years = command.getPeriod().getValue();
validateRegistrationPeriod(years);
@ -220,8 +223,6 @@ public final class DomainApplicationCreateFlow implements TransactionalFlow {
LaunchCreateExtension launchCreate =
eppInput.getSingleExtension(LaunchCreateExtension.class).get();
validateLaunchCreateExtension(launchCreate, registry, domainName, now);
boolean isAnchorTenant =
isAnchorTenant(domainName, Optional.empty(), authInfo.getPw().getValue(), Optional.empty());
// Superusers can create reserved domains, force creations on domains that require a claims
// notice without specifying a claims key, and override blocks on registering premium domains.
if (!isSuperuser) {

22
java/google/registry/flows/domain/DomainCreateFlow.java
View file

@ -23,6 +23,7 @@ import static google.registry.flows.domain.DomainFlowUtils.cloneAndLinkReference
import static google.registry.flows.domain.DomainFlowUtils.createFeeCreateResponse;
import static google.registry.flows.domain.DomainFlowUtils.getReservationTypes;
import static google.registry.flows.domain.DomainFlowUtils.isAnchorTenant;
import static google.registry.flows.domain.DomainFlowUtils.isReserved;
import static google.registry.flows.domain.DomainFlowUtils.isValidReservedCreate;
import static google.registry.flows.domain.DomainFlowUtils.validateCreateCommandContactsAndNameservers;
import static google.registry.flows.domain.DomainFlowUtils.validateDomainAllowedOnCreateRestrictedTld;
@ -113,7 +114,7 @@ import google.registry.model.reporting.DomainTransactionRecord;
import google.registry.model.reporting.DomainTransactionRecord.TransactionReportField;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.tmch.LordnTask;
import google.registry.tmch.LordnTaskUtils;
import java.util.Optional;
import javax.inject.Inject;
import org.joda.time.DateTime;
@ -308,7 +309,8 @@ public class DomainCreateFlow implements TransactionalFlow {
.build());
Optional<FeeCreateCommandExtension> feeCreate =
eppInput.getSingleExtension(FeeCreateCommandExtension.class);
FeesAndCredits feesAndCredits = pricingLogic.getCreatePrice(registry, targetId, now, years);
FeesAndCredits feesAndCredits =
pricingLogic.getCreatePrice(registry, targetId, now, years, isAnchorTenant);
validateFeeChallenge(targetId, registry.getTldStr(), clientId, now, feeCreate, feesAndCredits);
Optional<SecDnsCreateExtension> secDnsCreate =
validateSecDnsExtension(eppInput.getSingleExtension(SecDnsCreateExtension.class));
@ -319,7 +321,14 @@ public class DomainCreateFlow implements TransactionalFlow {
// Bill for the create.
BillingEvent.OneTime createBillingEvent =
createOneTimeBillingEvent(
registry, isAnchorTenant, isSunriseCreate, years, feesAndCredits, historyEntry, now);
registry,
isAnchorTenant,
isSunriseCreate,
isReserved(domainName, isSunriseCreate),
years,
feesAndCredits,
historyEntry,
now);
// Create a new autorenew billing event and poll message starting at the expiration time.
BillingEvent.Recurring autorenewBillingEvent =
createAutorenewBillingEvent(historyEntry, registrationExpirationTime);
@ -506,6 +515,7 @@ public class DomainCreateFlow implements TransactionalFlow {
Registry registry,
boolean isAnchorTenant,
boolean isSunriseCreate,
boolean isReserved,
int years,
FeesAndCredits feesAndCredits,
HistoryEntry historyEntry,
@ -517,6 +527,10 @@ public class DomainCreateFlow implements TransactionalFlow {
}
if (isAnchorTenant) {
flagsBuilder.add(Flag.ANCHOR_TENANT);
} else if (isReserved) {
// Don't add this flag if the domain is an anchor tenant (which are also reserved); only add
// it if it's reserved for other reasons.
flagsBuilder.add(Flag.RESERVED);
}
return new BillingEvent.OneTime.Builder()
.setReason(Reason.CREATE)
@ -594,7 +608,7 @@ public class DomainCreateFlow implements TransactionalFlow {
dnsQueue.addDomainRefreshTask(newDomain.getFullyQualifiedDomainName());
}
if (hasClaimsNotice || hasSignedMarks) {
LordnTask.enqueueDomainResourceTask(newDomain);
LordnTaskUtils.enqueueDomainResourceTask(newDomain);
}
}

16
java/google/registry/flows/domain/DomainFlowUtils.java
View file

@ -32,7 +32,6 @@ import static google.registry.model.registry.label.ReservationType.RESERVED_FOR_
import static google.registry.model.registry.label.ReservationType.RESERVED_FOR_SPECIFIC_USE;
import static google.registry.model.registry.label.ReservedList.getAllowedNameservers;
import static google.registry.pricing.PricingEngineProxy.isDomainPremium;
import static google.registry.tldconfig.idn.IdnLabelValidator.findValidIdnTableForTld;
import static google.registry.util.CollectionUtils.nullToEmpty;
import static google.registry.util.DateTimeUtils.END_OF_TIME;
import static google.registry.util.DateTimeUtils.isAtOrAfter;
@ -113,6 +112,7 @@ import google.registry.model.reporting.DomainTransactionRecord;
import google.registry.model.reporting.DomainTransactionRecord.TransactionReportField;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.tmch.ClaimsListShard;
import google.registry.tldconfig.idn.IdnLabelValidator;
import google.registry.util.Idn;
import java.math.BigDecimal;
import java.util.Comparator;
@ -166,6 +166,10 @@ public class DomainFlowUtils {
private static final CharMatcher ALLOWED_CHARS =
CharMatcher.inRange('a', 'z').or(CharMatcher.inRange('0', '9').or(CharMatcher.anyOf("-.")));
/** Default validator used to determine if an IDN name can be provisioned on a TLD. */
private static final IdnLabelValidator IDN_LABEL_VALIDATOR =
IdnLabelValidator.createDefaultIdnLabelValidator();
/** The maximum number of DS records allowed on a domain. */
private static final int MAX_DS_RECORDS_PER_DOMAIN = 8;
@ -239,7 +243,8 @@ public class DomainFlowUtils {
public static String validateDomainNameWithIdnTables(InternetDomainName domainName)
throws InvalidIdnDomainLabelException {
Optional<String> idnTableName =
findValidIdnTableForTld(domainName.parts().get(0), domainName.parent().toString());
IDN_LABEL_VALIDATOR.findValidIdnTableForTld(
domainName.parts().get(0), domainName.parent().toString());
if (!idnTableName.isPresent()) {
throw new InvalidIdnDomainLabelException();
}
@ -453,7 +458,7 @@ public class DomainFlowUtils {
private static final ImmutableSet<ReservationType> RESERVED_TYPES =
ImmutableSet.of(RESERVED_FOR_SPECIFIC_USE, RESERVED_FOR_ANCHOR_TENANT, FULLY_BLOCKED);
private static boolean isReserved(InternetDomainName domainName, boolean isSunrise) {
static boolean isReserved(InternetDomainName domainName, boolean isSunrise) {
ImmutableSet<ReservationType> types = getReservationTypes(domainName);
return !Sets.intersection(types, RESERVED_TYPES).isEmpty()
|| !(isSunrise || intersection(TYPES_ALLOWED_FOR_CREATE_ONLY_IN_SUNRISE, types).isEmpty());
@ -621,7 +626,10 @@ public class DomainFlowUtils {
builder.setReasonIfSupported("reserved");
} else {
builder.setAvailIfSupported(true);
fees = pricingLogic.getCreatePrice(registry, domainNameString, now, years).getFees();
// TODO(b/117145844): Once allocation token support for domain check flow is implemented,
// we should be able to calculate the correct price here.
fees =
pricingLogic.getCreatePrice(registry, domainNameString, now, years, false).getFees();
}
break;
case RENEW:

6
java/google/registry/flows/domain/DomainPricingLogic.java
View file

@ -54,7 +54,8 @@ public final class DomainPricingLogic {
/** Returns a new create price for the pricer. */
public FeesAndCredits getCreatePrice(
Registry registry, String domainName, DateTime date, int years) throws EppException {
Registry registry, String domainName, DateTime date, int years, boolean isAnchorTenant)
throws EppException {
CurrencyUnit currency = registry.getCurrency();
// Get the vanilla create cost.
@ -65,7 +66,8 @@ public final class DomainPricingLogic {
Fee eapFee = registry.getEapFeeFor(date);
FeesAndCredits.Builder feesBuilder =
new FeesAndCredits.Builder().setCurrency(currency).addFeeOrCredit(createFeeOrCredit);
if (!eapFee.hasZeroCost()) {
// Don't charge anchor tenants EAP fees.
if (!isAnchorTenant && !eapFee.hasZeroCost()) {
feesBuilder.addFeeOrCredit(eapFee);
}

21
java/google/registry/keyring/BUILD Normal file
View file

@ -0,0 +1,21 @@
package(
default_visibility = ["//visibility:public"],
)
licenses(["notice"]) # Apache 2.0
java_library(
name = "keyring",
srcs = glob(["*.java"]),
deps = [
"//java/google/registry/config",
"//java/google/registry/keyring/api",
"@com_google_code_findbugs_jsr305",
"@com_google_dagger",
"@com_google_flogger",
"@com_google_flogger_system_backend",
"@com_google_guava",
"@javax_inject",
"@org_bouncycastle_bcpg_jdk15on",
],
)

18
java/google/registry/keyring/kms/KeyringModule.java → java/google/registry/keyring/KeyringModule.java
View file

@ -1,4 +1,4 @@
// Copyright 2017 The Nomulus Authors. All Rights Reserved.
// Copyright 2018 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,11 +12,15 @@
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.keyring.kms;
package google.registry.keyring;
import static com.google.common.base.Preconditions.checkState;
import dagger.Module;
import dagger.Provides;
import google.registry.config.RegistryConfig.Config;
import google.registry.keyring.api.Keyring;
import java.util.Map;
import javax.inject.Singleton;
/** Dagger module for {@link Keyring} */
@ -25,7 +29,13 @@ public final class KeyringModule {
@Provides
@Singleton
public static Keyring provideKeyring(KmsKeyring kmsKeyring) {
return kmsKeyring;
public static Keyring provideKeyring(
Map<String, Keyring> keyrings, @Config("activeKeyring") String activeKeyring) {
checkState(
keyrings.containsKey(activeKeyring),
"Invalid Keyring %s is configured; valid choices are %s",
activeKeyring,
keyrings.keySet());
return keyrings.get(activeKeyring);
}
}

18
java/google/registry/keyring/api/DummyKeyringModule.java
View file

@ -21,11 +21,15 @@ import static google.registry.keyring.api.PgpHelper.lookupKeyPair;
import com.google.common.base.VerifyException;
import com.google.common.io.ByteSource;
import com.google.common.io.Resources;
import dagger.Binds;
import dagger.Module;
import dagger.Provides;
import dagger.multibindings.IntoMap;
import dagger.multibindings.StringKey;
import java.io.IOException;
import java.io.InputStream;
import javax.annotation.concurrent.Immutable;
import javax.inject.Named;
import org.bouncycastle.openpgp.PGPException;
import org.bouncycastle.openpgp.PGPKeyPair;
import org.bouncycastle.openpgp.PGPPublicKeyRingCollection;
@ -68,7 +72,9 @@ import org.bouncycastle.openpgp.bc.BcPGPSecretKeyRingCollection;
*/
@Module
@Immutable
public final class DummyKeyringModule {
public abstract class DummyKeyringModule {
public static final String NAME = "Dummy";
/** The contents of a dummy PGP public key stored in a file. */
private static final ByteSource PGP_PUBLIC_KEYRING =
@ -81,9 +87,15 @@ public final class DummyKeyringModule {
/** The email address of the aforementioned PGP key. */
private static final String EMAIL_ADDRESS = "test-registry@example.com";
@Binds
@IntoMap
@StringKey(NAME)
abstract Keyring provideKeyring(@Named("DummyKeyring") InMemoryKeyring keyring);
/** Always returns a {@link InMemoryKeyring} instance. */
@Provides
static Keyring provideKeyring() {
@Named("DummyKeyring")
static InMemoryKeyring provideDummyKeyring() {
PGPKeyPair dummyKey;
try (InputStream publicInput = PGP_PUBLIC_KEYRING.openStream();
InputStream privateInput = PGP_PRIVATE_KEYRING.openStream()) {
@ -112,4 +124,6 @@ public final class DummyKeyringModule {
"not a real login",
"not a real credential");
}
private DummyKeyringModule() {}
}

12
java/google/registry/keyring/kms/KmsModule.java
View file

@ -19,13 +19,23 @@ import com.google.api.services.cloudkms.v1.CloudKMS;
import dagger.Binds;
import dagger.Module;
import dagger.Provides;
import dagger.multibindings.IntoMap;
import dagger.multibindings.StringKey;
import google.registry.config.CredentialModule.DefaultCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.keyring.api.Keyring;
/** Dagger module for Cloud KMS connection objects. */
/** Dagger module for Cloud KMS. */
@Module
public abstract class KmsModule {
public static final String NAME = "KMS";
@Binds
@IntoMap
@StringKey(NAME)
abstract Keyring provideKeyring(KmsKeyring keyring);
@Provides
static CloudKMS provideKms(
@DefaultCredential GoogleCredential credential,

8
java/google/registry/model/billing/BillingEvent.java
View file

@ -71,6 +71,14 @@ public abstract class BillingEvent extends ImmutableObject
ANCHOR_TENANT,
AUTO_RENEW,
LANDRUSH,
/**
* This flag is used on create {@link OneTime} billing events for domains that were reserved.
*
* <p>This can happen when allocation tokens are used or superusers override a domain
* reservation. These cases can need special handling in billing/invoicing. Anchor tenants will
* never have this flag applied; they will have ANCHOR_TENANT instead.
*/
RESERVED,
SUNRISE,
/**
* This flag will be added to any {@link OneTime} events that are created via, e.g., an

2
java/google/registry/model/domain/token/AllocationToken.java
View file

@ -41,7 +41,7 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
@Id String token;
/** The key of the history entry for which the token was used. Null if not yet used. */
Key<HistoryEntry> redemptionHistoryEntry;
@Nullable @Index Key<HistoryEntry> redemptionHistoryEntry;
/** The fully-qualified domain name that this token is limited to, if any. */
@Nullable @Index String domainName;

12
java/google/registry/model/tmch/ClaimsListShard.java
View file

@ -39,7 +39,6 @@ import google.registry.model.annotations.VirtualEntity;
import google.registry.model.common.CrossTldSingleton;
import google.registry.util.CollectionUtils;
import google.registry.util.Concurrent;
import google.registry.util.NonFinalForTesting;
import google.registry.util.Retrier;
import google.registry.util.SystemSleeper;
import java.util.HashMap;
@ -69,10 +68,8 @@ import org.joda.time.DateTime;
@NotBackedUp(reason = Reason.EXTERNALLY_SOURCED)
public class ClaimsListShard extends ImmutableObject {
/** The number of claims list entries to store per shard. Do not modify except for in tests. */
@VisibleForTesting
@NonFinalForTesting
static int shardSize = 10000;
/** The number of claims list entries to store per shard. */
private static final int SHARD_SIZE = 10000;
@Id
long id;
@ -165,6 +162,11 @@ public class ClaimsListShard extends ImmutableObject {
* switching over to using them atomically, then deleting the old ones.
*/
public void save() {
save(SHARD_SIZE);
}
@VisibleForTesting
void save(int shardSize) {
// Figure out what the next versionId should be based on which ones already exist.
final Key<ClaimsListRevision> oldRevision = getCurrentRevision();
final Key<ClaimsListRevision> parentKey = ClaimsListRevision.createKey();

1
java/google/registry/module/backend/BUILD
View file

@ -22,6 +22,7 @@ java_library(
"//java/google/registry/flows",
"//java/google/registry/gcs",
"//java/google/registry/groups",
"//java/google/registry/keyring",
"//java/google/registry/keyring/api",
"//java/google/registry/keyring/kms",
"//java/google/registry/mapreduce",

11
java/google/registry/module/backend/BackendComponent.java
View file

@ -27,19 +27,18 @@ import google.registry.gcs.GcsServiceModule;
import google.registry.groups.DirectoryModule;
import google.registry.groups.GroupsModule;
import google.registry.groups.GroupssettingsModule;
import google.registry.keyring.KeyringModule;
import google.registry.keyring.api.DummyKeyringModule;
import google.registry.keyring.api.KeyModule;
import google.registry.keyring.kms.KmsModule;
import google.registry.module.backend.BackendRequestComponent.BackendRequestComponentModule;
import google.registry.monitoring.whitebox.StackdriverModule;
import google.registry.rde.JSchModule;
import google.registry.request.Modules.AppIdentityCredentialModule;
import google.registry.request.Modules.DatastoreServiceModule;
import google.registry.request.Modules.GoogleCredentialModule;
import google.registry.request.Modules.Jackson2Module;
import google.registry.request.Modules.NetHttpTransportModule;
import google.registry.request.Modules.URLFetchServiceModule;
import google.registry.request.Modules.UrlFetchTransportModule;
import google.registry.request.Modules.UseAppIdentityCredentialForGoogleApisModule;
import google.registry.request.Modules.UserServiceModule;
import google.registry.request.auth.AuthModule;
import google.registry.util.AppEngineServiceUtilsImpl.AppEngineServiceUtilsModule;
@ -52,7 +51,6 @@ import javax.inject.Singleton;
@Component(
modules = {
AppEngineServiceUtilsModule.class,
AppIdentityCredentialModule.class,
AuthModule.class,
BackendRequestComponentModule.class,
BigqueryModule.class,
@ -60,15 +58,15 @@ import javax.inject.Singleton;
CredentialModule.class,
DatastoreServiceModule.class,
DirectoryModule.class,
google.registry.keyring.api.DummyKeyringModule.class,
DummyKeyringModule.class,
DriveModule.class,
GcsServiceModule.class,
GoogleCredentialModule.class,
GroupsModule.class,
GroupssettingsModule.class,
JSchModule.class,
Jackson2Module.class,
KeyModule.class,
KeyringModule.class,
KmsModule.class,
NetHttpTransportModule.class,
SheetsServiceModule.class,
@ -77,7 +75,6 @@ import javax.inject.Singleton;
SystemSleeperModule.class,
URLFetchServiceModule.class,
UrlFetchTransportModule.class,
UseAppIdentityCredentialForGoogleApisModule.class,
UserServiceModule.class,
VoidDnsWriterModule.class,
})

2
java/google/registry/module/backend/BackendRequestComponent.java
View file

@ -53,7 +53,6 @@ import google.registry.export.sheet.SheetModule;
import google.registry.export.sheet.SyncRegistrarsSheetAction;
import google.registry.flows.async.AsyncFlowsModule;
import google.registry.mapreduce.MapreduceModule;
import google.registry.monitoring.whitebox.MetricsExportAction;
import google.registry.monitoring.whitebox.WhiteboxModule;
import google.registry.rde.BrdaCopyAction;
import google.registry.rde.RdeModule;
@ -136,7 +135,6 @@ interface BackendRequestComponent {
IcannReportingStagingAction icannReportingStagingAction();
IcannReportingUploadAction icannReportingUploadAction();
LoadSnapshotAction loadSnapshotAction();
MetricsExportAction metricsExportAction();
NordnUploadAction nordnUploadAction();
NordnVerifyAction nordnVerifyAction();
PublishDnsUpdatesAction publishDnsUpdatesAction();

1
java/google/registry/module/frontend/BUILD
View file

@ -11,6 +11,7 @@ java_library(
"//java/google/registry/config",
"//java/google/registry/dns",
"//java/google/registry/flows",
"//java/google/registry/keyring",
"//java/google/registry/keyring/api",
"//java/google/registry/keyring/kms",
"//java/google/registry/monitoring/whitebox",

11
java/google/registry/module/frontend/FrontendComponent.java
View file

@ -21,16 +21,15 @@ import google.registry.config.CredentialModule;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.flows.ServerTridProviderModule;
import google.registry.flows.custom.CustomLogicFactoryModule;
import google.registry.keyring.KeyringModule;
import google.registry.keyring.api.DummyKeyringModule;
import google.registry.keyring.api.KeyModule;
import google.registry.keyring.kms.KmsModule;
import google.registry.module.frontend.FrontendRequestComponent.FrontendRequestComponentModule;
import google.registry.monitoring.whitebox.StackdriverModule;
import google.registry.request.Modules.AppIdentityCredentialModule;
import google.registry.request.Modules.GoogleCredentialModule;
import google.registry.request.Modules.Jackson2Module;
import google.registry.request.Modules.NetHttpTransportModule;
import google.registry.request.Modules.UrlFetchTransportModule;
import google.registry.request.Modules.UseAppIdentityCredentialForGoogleApisModule;
import google.registry.request.Modules.UserServiceModule;
import google.registry.request.auth.AuthModule;
import google.registry.ui.ConsoleDebug.ConsoleConfigModule;
@ -44,17 +43,16 @@ import javax.inject.Singleton;
@Component(
modules = {
AppEngineServiceUtilsModule.class,
AppIdentityCredentialModule.class,
AuthModule.class,
ConfigModule.class,
ConsoleConfigModule.class,
CredentialModule.class,
CustomLogicFactoryModule.class,
google.registry.keyring.api.DummyKeyringModule.class,
DummyKeyringModule.class,
FrontendRequestComponentModule.class,
GoogleCredentialModule.class,
Jackson2Module.class,
KeyModule.class,
KeyringModule.class,
KmsModule.class,
NetHttpTransportModule.class,
ServerTridProviderModule.class,
@ -62,7 +60,6 @@ import javax.inject.Singleton;
SystemClockModule.class,
SystemSleeperModule.class,
UrlFetchTransportModule.class,
UseAppIdentityCredentialForGoogleApisModule.class,
UserServiceModule.class,
})
interface FrontendComponent {

2
java/google/registry/module/frontend/FrontendRequestComponent.java
View file

@ -26,12 +26,14 @@ import google.registry.request.RequestComponentBuilder;
import google.registry.request.RequestModule;
import google.registry.request.RequestScope;
import google.registry.ui.server.registrar.ConsoleUiAction;
import google.registry.ui.server.registrar.RegistrarConsoleModule;
import google.registry.ui.server.registrar.RegistrarSettingsAction;
/** Dagger component with per-request lifetime for "default" App Engine module. */
@RequestScope
@Subcomponent(
modules = {
RegistrarConsoleModule.class,
DnsModule.class,
EppTlsModule.class,
RequestModule.class,

1
java/google/registry/module/pubapi/BUILD
View file

@ -11,6 +11,7 @@ java_library(
"//java/google/registry/config",
"//java/google/registry/dns",
"//java/google/registry/flows",
"//java/google/registry/keyring",
"//java/google/registry/keyring/api",
"//java/google/registry/keyring/kms",
"//java/google/registry/monitoring/whitebox",

11
java/google/registry/module/pubapi/PubApiComponent.java
View file

@ -21,16 +21,15 @@ import google.registry.config.CredentialModule;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.flows.ServerTridProviderModule;
import google.registry.flows.custom.CustomLogicFactoryModule;
import google.registry.keyring.KeyringModule;
import google.registry.keyring.api.DummyKeyringModule;
import google.registry.keyring.api.KeyModule;
import google.registry.keyring.kms.KmsModule;
import google.registry.module.pubapi.PubApiRequestComponent.PubApiRequestComponentModule;
import google.registry.monitoring.whitebox.StackdriverModule;
import google.registry.request.Modules.AppIdentityCredentialModule;
import google.registry.request.Modules.GoogleCredentialModule;
import google.registry.request.Modules.Jackson2Module;
import google.registry.request.Modules.NetHttpTransportModule;
import google.registry.request.Modules.UrlFetchTransportModule;
import google.registry.request.Modules.UseAppIdentityCredentialForGoogleApisModule;
import google.registry.request.Modules.UserServiceModule;
import google.registry.request.auth.AuthModule;
import google.registry.util.AppEngineServiceUtilsImpl.AppEngineServiceUtilsModule;
@ -43,16 +42,15 @@ import javax.inject.Singleton;
@Component(
modules = {
AppEngineServiceUtilsModule.class,
AppIdentityCredentialModule.class,
AuthModule.class,
ConfigModule.class,
CredentialModule.class,
CustomLogicFactoryModule.class,
google.registry.keyring.api.DummyKeyringModule.class,
DummyKeyringModule.class,
PubApiRequestComponentModule.class,
GoogleCredentialModule.class,
Jackson2Module.class,
KeyModule.class,
KeyringModule.class,
KmsModule.class,
NetHttpTransportModule.class,
ServerTridProviderModule.class,
@ -60,7 +58,6 @@ import javax.inject.Singleton;
SystemClockModule.class,
SystemSleeperModule.class,
UrlFetchTransportModule.class,
UseAppIdentityCredentialForGoogleApisModule.class,
UserServiceModule.class,
})
interface PubApiComponent {

1
java/google/registry/module/tools/BUILD
View file

@ -15,6 +15,7 @@ java_library(
"//java/google/registry/flows",
"//java/google/registry/gcs",
"//java/google/registry/groups",
"//java/google/registry/keyring",
"//java/google/registry/keyring/api",
"//java/google/registry/keyring/kms",
"//java/google/registry/loadtest",

11
java/google/registry/module/tools/ToolsComponent.java
View file

@ -24,16 +24,15 @@ import google.registry.gcs.GcsServiceModule;
import google.registry.groups.DirectoryModule;
import google.registry.groups.GroupsModule;
import google.registry.groups.GroupssettingsModule;
import google.registry.keyring.KeyringModule;
import google.registry.keyring.api.DummyKeyringModule;
import google.registry.keyring.api.KeyModule;
import google.registry.keyring.kms.KmsModule;
import google.registry.module.tools.ToolsRequestComponent.ToolsRequestComponentModule;
import google.registry.request.Modules.AppIdentityCredentialModule;
import google.registry.request.Modules.DatastoreServiceModule;
import google.registry.request.Modules.GoogleCredentialModule;
import google.registry.request.Modules.Jackson2Module;
import google.registry.request.Modules.NetHttpTransportModule;
import google.registry.request.Modules.UrlFetchTransportModule;
import google.registry.request.Modules.UseAppIdentityCredentialForGoogleApisModule;
import google.registry.request.Modules.UserServiceModule;
import google.registry.request.auth.AuthModule;
import google.registry.util.AppEngineServiceUtilsImpl.AppEngineServiceUtilsModule;
@ -46,21 +45,20 @@ import javax.inject.Singleton;
@Component(
modules = {
AppEngineServiceUtilsModule.class,
AppIdentityCredentialModule.class,
AuthModule.class,
ConfigModule.class,
CredentialModule.class,
CustomLogicFactoryModule.class,
DatastoreServiceModule.class,
DirectoryModule.class,
google.registry.keyring.api.DummyKeyringModule.class,
DummyKeyringModule.class,
DriveModule.class,
GcsServiceModule.class,
GoogleCredentialModule.class,
GroupsModule.class,
GroupssettingsModule.class,
Jackson2Module.class,
KeyModule.class,
KeyringModule.class,
KmsModule.class,
NetHttpTransportModule.class,
ServerTridProviderModule.class,
@ -68,7 +66,6 @@ import javax.inject.Singleton;
SystemSleeperModule.class,
ToolsRequestComponentModule.class,
UrlFetchTransportModule.class,
UseAppIdentityCredentialForGoogleApisModule.class,
UserServiceModule.class,
})
interface ToolsComponent {

36
java/google/registry/monitoring/whitebox/BigQueryMetric.java
View file

@ -1,36 +0,0 @@
// Copyright 2017 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.monitoring.whitebox;
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
/**
* A metric which can be encoded into a BigQuery row.
*
* @see BigQueryMetricsEnqueuer
*/
public interface BigQueryMetric {
/** Get the BigQuery table name for this metric. */
String getTableId();
/** Get the schema description for the BigQuery table. */
ImmutableList<TableFieldSchema> getSchemaFields();
/** Get a map of the row values for this metric instance. */
ImmutableMap<String, String> getBigQueryRowEncoding();
}

65
java/google/registry/monitoring/whitebox/BigQueryMetricsEnqueuer.java
View file

@ -1,65 +0,0 @@
// Copyright 2017 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.monitoring.whitebox;
import static com.google.appengine.api.taskqueue.TaskOptions.Builder.withUrl;
import com.google.appengine.api.taskqueue.Queue;
import com.google.appengine.api.taskqueue.TaskOptions;
import com.google.appengine.api.taskqueue.TransientFailureException;
import com.google.common.base.Supplier;
import com.google.common.flogger.FluentLogger;
import google.registry.util.AppEngineServiceUtils;
import java.util.Map.Entry;
import javax.inject.Inject;
import javax.inject.Named;
/**
* A collector of metric information. Enqueues collected metrics to a task queue to be written to
* BigQuery asynchronously.
*
* @see MetricsExportAction
*/
public class BigQueryMetricsEnqueuer {
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
public static final String QUEUE_BIGQUERY_STREAMING_METRICS = "bigquery-streaming-metrics";
@Inject AppEngineServiceUtils appEngineServiceUtils;
@Inject @Named("insertIdGenerator") Supplier<String> idGenerator;
@Inject @Named(QUEUE_BIGQUERY_STREAMING_METRICS) Queue queue;
@Inject BigQueryMetricsEnqueuer() {}
public void export(BigQueryMetric metric) {
try {
String hostname = appEngineServiceUtils.getCurrentVersionHostname("backend");
TaskOptions opts =
withUrl(MetricsExportAction.PATH)
.header("Host", hostname)
.param("insertId", idGenerator.get());
for (Entry<String, String> entry : metric.getBigQueryRowEncoding().entrySet()) {
opts.param(entry.getKey(), entry.getValue());
}
opts.param("tableId", metric.getTableId());
queue.add(opts);
} catch (TransientFailureException e) {
// Log and swallow. We may drop some metrics here but this should be rare.
logger.atInfo().withCause(e).log(
"Transient error occurred while recording metric; metric dropped.");
}
}
}

96
java/google/registry/monitoring/whitebox/EppMetric.java
View file

@ -15,44 +15,19 @@
package google.registry.monitoring.whitebox;
import static com.google.common.base.Preconditions.checkArgument;
import static google.registry.bigquery.BigqueryUtils.toBigqueryTimestamp;
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.auto.value.AutoValue;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Iterables;
import google.registry.bigquery.BigqueryUtils.FieldType;
import google.registry.model.eppoutput.Result.Code;
import google.registry.model.registry.Registries;
import google.registry.util.Clock;
import java.util.Optional;
import org.joda.time.DateTime;
/**
* A value class for recording attributes of an EPP metric.
*
* @see BigQueryMetricsEnqueuer
*/
/** A value class for recording attributes of an EPP metric. */
@AutoValue
public abstract class EppMetric implements BigQueryMetric {
static final String TABLE_ID = "eppMetrics";
static final ImmutableList<TableFieldSchema> SCHEMA_FIELDS =
ImmutableList.of(
new TableFieldSchema().setName("requestId").setType(FieldType.STRING.name()),
new TableFieldSchema().setName("startTime").setType(FieldType.TIMESTAMP.name()),
new TableFieldSchema().setName("endTime").setType(FieldType.TIMESTAMP.name()),
new TableFieldSchema().setName("commandName").setType(FieldType.STRING.name()),
new TableFieldSchema().setName("clientId").setType(FieldType.STRING.name()),
new TableFieldSchema().setName("tld").setType(FieldType.STRING.name()),
new TableFieldSchema().setName("privilegeLevel").setType(FieldType.STRING.name()),
new TableFieldSchema().setName("eppTarget").setType(FieldType.STRING.name()),
new TableFieldSchema().setName("eppStatus").setType(FieldType.INTEGER.name()),
new TableFieldSchema().setName("attempts").setType(FieldType.INTEGER.name()));
public abstract String getRequestId();
public abstract class EppMetric {
public abstract DateTime getStartTimestamp();
@ -64,55 +39,8 @@ public abstract class EppMetric implements BigQueryMetric {
public abstract Optional<String> getTld();
public abstract Optional<String> getPrivilegeLevel();
public abstract Optional<String> getEppTarget();
public abstract Optional<Code> getStatus();
public abstract Integer getAttempts();
@Override
public String getTableId() {
return TABLE_ID;
}
@Override
public ImmutableList<TableFieldSchema> getSchemaFields() {
return SCHEMA_FIELDS;
}
@Override
public ImmutableMap<String, String> getBigQueryRowEncoding() {
// Create map builder, start with required values
ImmutableMap.Builder<String, String> map =
new ImmutableMap.Builder<String, String>()
.put("requestId", getRequestId())
.put("startTime", toBigqueryTimestamp(getStartTimestamp()))
.put("endTime", toBigqueryTimestamp(getEndTimestamp()))
.put("attempts", getAttempts().toString());
// Populate optional values, if present
addOptional("commandName", getCommandName(), map);
addOptional("clientId", getClientId(), map);
addOptional("tld", getTld(), map);
addOptional("privilegeLevel", getPrivilegeLevel(), map);
addOptional("eppTarget", getEppTarget(), map);
if (getStatus().isPresent()) {
map.put("eppStatus", Integer.toString(getStatus().get().code));
}
return map.build();
}
/**
* Helper method to populate an {@link com.google.common.collect.ImmutableMap.Builder} with an
* {@link Optional} value if the value is {@link Optional#isPresent()}.
*/
private static <T> void addOptional(
String key, Optional<T> value, ImmutableMap.Builder<String, String> map) {
value.ifPresent(t -> map.put(key, t.toString()));
}
/** Create an {@link EppMetric.Builder}. */
public static Builder builder() {
return new AutoValue_EppMetric.Builder();
@ -124,9 +52,8 @@ public abstract class EppMetric implements BigQueryMetric {
*
* <p>The start timestamp is recorded now, and the end timestamp at {@code build()}.
*/
public static Builder builderForRequest(String requestId, Clock clock) {
public static Builder builderForRequest(Clock clock) {
return builder()
.setRequestId(requestId)
.setStartTimestamp(clock.nowUtc())
.setClock(clock);
}
@ -135,14 +62,9 @@ public abstract class EppMetric implements BigQueryMetric {
@AutoValue.Builder
public abstract static class Builder {
/** Builder-only counter of the number of attempts, to support {@link #incrementAttempts()}. */
private int attempts = 0;
/** Builder-only clock to support automatic recording of endTimestamp on {@link #build()}. */
private Clock clock = null;
abstract Builder setRequestId(String requestId);
abstract Builder setStartTimestamp(DateTime startTimestamp);
abstract Builder setEndTimestamp(DateTime endTimestamp);
@ -191,19 +113,8 @@ public abstract class EppMetric implements BigQueryMetric {
return this;
}
public abstract Builder setPrivilegeLevel(String privilegeLevel);
public abstract Builder setEppTarget(String eppTarget);
public abstract Builder setStatus(Code code);
abstract Builder setAttempts(Integer attempts);
public Builder incrementAttempts() {
attempts++;
return this;
}
Builder setClock(Clock clock) {
this.clock = clock;
return this;
@ -216,7 +127,6 @@ public abstract class EppMetric implements BigQueryMetric {
* current timestamp of the clock; otherwise end timestamp must have been previously set.
*/
public EppMetric build() {
setAttempts(attempts);
if (clock != null) {
setEndTimestamp(clock.nowUtc());
}

104
java/google/registry/monitoring/whitebox/MetricsExportAction.java
View file

@ -1,104 +0,0 @@
// Copyright 2017 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.monitoring.whitebox;
import static com.google.common.base.Predicates.in;
import static com.google.common.base.Predicates.not;
import static com.google.common.collect.Multimaps.filterKeys;
import static google.registry.request.Action.Method.POST;
import static java.util.stream.Collectors.joining;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.model.TableDataInsertAllRequest;
import com.google.api.services.bigquery.model.TableDataInsertAllResponse;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableListMultimap;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.flogger.FluentLogger;
import google.registry.bigquery.CheckedBigquery;
import google.registry.config.RegistryConfig.Config;
import google.registry.request.Action;
import google.registry.request.Parameter;
import google.registry.request.ParameterMap;
import google.registry.request.auth.Auth;
import java.io.IOException;
import java.util.Map;
import javax.inject.Inject;
/** Action for exporting metrics to BigQuery. */
@Action(
path = MetricsExportAction.PATH,
method = POST,
auth = Auth.AUTH_INTERNAL_ONLY
)
public class MetricsExportAction implements Runnable {
public static final String PATH = "/_dr/task/metrics";
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private static final String DATASET_ID = "metrics";
private static final ImmutableSet<String> SPECIAL_PARAMS = ImmutableSet.of("tableId", "insertId");
@Inject @Parameter("tableId") String tableId;
@Inject @Parameter("insertId") String insertId;
@Inject @Config("projectId") String projectId;
@Inject CheckedBigquery checkedBigquery;
@Inject @ParameterMap ImmutableListMultimap<String, String> parameters;
@Inject MetricsExportAction() {}
/** Exports metrics to BigQuery. */
@Override
public void run() {
try {
Bigquery bigquery =
checkedBigquery.ensureDataSetAndTableExist(projectId, DATASET_ID, tableId);
// Filter out the special parameters that the Action is called with. Everything that's left
// is returned in a Map that is suitable to pass to Bigquery as row data.
Map<String, Object> jsonRows =
ImmutableMap.copyOf(
filterKeys(parameters, not(in(SPECIAL_PARAMS))).entries());
TableDataInsertAllResponse response = bigquery.tabledata()
.insertAll(
projectId,
DATASET_ID,
tableId,
new TableDataInsertAllRequest()
.setRows(
ImmutableList.of(new TableDataInsertAllRequest.Rows()
.setInsertId(insertId)
.setJson(jsonRows))))
.execute();
if (response.getInsertErrors() != null && !response.getInsertErrors().isEmpty()) {
throw new RuntimeException(
response
.getInsertErrors()
.stream()
.map(
error -> {
try {
return error.toPrettyString();
} catch (IOException e) {
return error.toString();
}
})
.collect(joining("\n")));
}
} catch (Throwable e) {
logger.atWarning().withCause(e).log("Unknown error while exporting metrics to BigQuery.");
}
}
}

51
java/google/registry/monitoring/whitebox/WhiteboxModule.java
View file

@ -14,24 +14,9 @@
package google.registry.monitoring.whitebox;
import static com.google.appengine.api.taskqueue.QueueFactory.getQueue;
import static google.registry.monitoring.whitebox.BigQueryMetricsEnqueuer.QUEUE_BIGQUERY_STREAMING_METRICS;
import static google.registry.request.RequestParameters.extractRequiredParameter;
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.appengine.api.taskqueue.Queue;
import com.google.common.base.Supplier;
import com.google.common.collect.ImmutableList;
import dagger.Module;
import dagger.Provides;
import dagger.multibindings.IntoMap;
import dagger.multibindings.StringKey;
import google.registry.request.Parameter;
import google.registry.request.RequestLogId;
import google.registry.util.Clock;
import java.util.UUID;
import javax.inject.Named;
import javax.servlet.http.HttpServletRequest;
/**
* Dagger module for injecting common settings for Whitebox tasks.
@ -39,46 +24,14 @@ import javax.servlet.http.HttpServletRequest;
@Module
public class WhiteboxModule {
@Provides
@IntoMap
@StringKey(EppMetric.TABLE_ID)
static ImmutableList<TableFieldSchema> provideEppMetricsSchema() {
return EppMetric.SCHEMA_FIELDS;
}
@Provides
@Parameter("tableId")
static String provideTableId(HttpServletRequest req) {
return extractRequiredParameter(req, "tableId");
}
@Provides
@Parameter("insertId")
static String provideInsertId(HttpServletRequest req) {
return extractRequiredParameter(req, "insertId");
}
@Provides
@Named("insertIdGenerator")
static Supplier<String> provideInsertIdGenerator() {
return () -> UUID.randomUUID().toString();
}
/** Provides an EppMetric builder with the request ID and startTimestamp already initialized. */
@Provides
static EppMetric.Builder provideEppMetricBuilder(
@RequestLogId String requestLogId, Clock clock) {
return EppMetric.builderForRequest(requestLogId, clock);
static EppMetric.Builder provideEppMetricBuilder(Clock clock) {
return EppMetric.builderForRequest(clock);
}
@Provides
static CheckApiMetric.Builder provideCheckApiMetricBuilder(Clock clock) {
return CheckApiMetric.builder(clock);
}
@Provides
@Named(QUEUE_BIGQUERY_STREAMING_METRICS)
static Queue provideBigQueryStreamingMetricsQueue() {
return getQueue(QUEUE_BIGQUERY_STREAMING_METRICS);
}
}

4
java/google/registry/proxy/handler/QuotaHandler.java
View file

@ -156,7 +156,9 @@ public abstract class QuotaHandler extends ChannelInboundHandlerAdapter {
public void channelInactive(ChannelHandlerContext ctx) {
// If no reads occurred before the connection is inactive (for example when the handshake
// is not successful), no quota is leased and therefore no return is needed.
if (quotaResponse != null) {
// Note that the quota response can be a failure, in which case no token was leased to us from
// the token store. Consequently no return is necessary.
if (quotaResponse != null && quotaResponse.success()) {
Future<?> unusedFuture = quotaManager.releaseQuota(QuotaRebate.create(quotaResponse));
}
ctx.fireChannelInactive();

1
java/google/registry/proxy/terraform/modules/common.tf
View file

@ -1,3 +1,4 @@
provider "google" {
version = ">= 1.13.0"
project = "${var.proxy_project_name}"
}

28
java/google/registry/rdap/RdapActionBase.java
View file

@ -16,7 +16,6 @@ package google.registry.rdap;
import static com.google.common.base.Charsets.UTF_8;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkState;
import static com.google.common.net.HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.util.DateTimeUtils.END_OF_TIME;
@ -27,6 +26,7 @@ import static javax.servlet.http.HttpServletResponse.SC_OK;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.flogger.FluentLogger;
import com.google.common.net.MediaType;
import com.google.re2j.Pattern;
@ -48,7 +48,8 @@ import google.registry.request.RequestPath;
import google.registry.request.Response;
import google.registry.request.auth.AuthResult;
import google.registry.request.auth.UserAuthInfo;
import google.registry.ui.server.registrar.SessionUtils;
import google.registry.ui.server.registrar.AuthenticatedRegistrarAccessor;
import google.registry.util.Clock;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
@ -57,7 +58,6 @@ import java.util.List;
import java.util.Optional;
import javax.annotation.Nullable;
import javax.inject.Inject;
import javax.servlet.http.HttpServletRequest;
import org.joda.time.DateTime;
import org.json.simple.JSONValue;
@ -87,13 +87,13 @@ public abstract class RdapActionBase implements Runnable {
INCLUDE
}
@Inject HttpServletRequest request;
@Inject Response response;
@Inject Clock clock;
@Inject @RequestMethod Action.Method requestMethod;
@Inject @RequestPath String requestPath;
@Inject @FullServletPath String fullServletPath;
@Inject AuthResult authResult;
@Inject SessionUtils sessionUtils;
@Inject AuthenticatedRegistrarAccessor registrarAccessor;
@Inject RdapJsonFormatter rdapJsonFormatter;
@Inject @Parameter("registrar") Optional<String> registrarParam;
@Inject @Parameter("includeDeleted") Optional<Boolean> includeDeletedParam;
@ -200,15 +200,12 @@ public abstract class RdapActionBase implements Runnable {
if (userAuthInfo.isUserAdmin()) {
return RdapAuthorization.ADMINISTRATOR_AUTHORIZATION;
}
if (!sessionUtils.checkRegistrarConsoleLogin(request, userAuthInfo)) {
ImmutableSet<String> clientIds = registrarAccessor.getAllClientIdWithRoles().keySet();
if (clientIds.isEmpty()) {
logger.atWarning().log("Couldn't find registrar for User %s.", authResult.userIdForLogging());
return RdapAuthorization.PUBLIC_AUTHORIZATION;
}
String clientId = sessionUtils.getRegistrarClientId(request);
Optional<Registrar> registrar = Registrar.loadByClientIdCached(clientId);
if (!registrar.isPresent()) {
return RdapAuthorization.PUBLIC_AUTHORIZATION;
}
return RdapAuthorization.create(RdapAuthorization.Role.REGISTRAR, clientId);
return RdapAuthorization.create(RdapAuthorization.Role.REGISTRAR, clientIds);
}
/** Returns the registrar on which results should be filtered, or absent(). */
@ -238,14 +235,9 @@ public abstract class RdapActionBase implements Runnable {
if (userAuthInfo.isUserAdmin()) {
return true;
}
if (!sessionUtils.checkRegistrarConsoleLogin(request, userAuthInfo)) {
if (registrarAccessor.getAllClientIdWithRoles().isEmpty()) {
return false;
}
String clientId = sessionUtils.getRegistrarClientId(request);
checkState(
Registrar.loadByClientIdCached(clientId).isPresent(),
"Registrar with clientId %s doesn't exist",
clientId);
return true;
}

13
java/google/registry/rdap/RdapAuthorization.java
View file

@ -15,7 +15,7 @@
package google.registry.rdap;
import com.google.auto.value.AutoValue;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import google.registry.model.ImmutableObject;
/** Authorization information for RDAP data access. */
@ -32,13 +32,13 @@ public abstract class RdapAuthorization extends ImmutableObject {
public abstract Role role();
/** The registrar client IDs for which access is granted (used only if the role is REGISTRAR. */
public abstract ImmutableList<String> clientIds();
public abstract ImmutableSet<String> clientIds();
static RdapAuthorization create(Role role, String clientId) {
return new AutoValue_RdapAuthorization(role, ImmutableList.of(clientId));
return new AutoValue_RdapAuthorization(role, ImmutableSet.of(clientId));
}
static RdapAuthorization create(Role role, ImmutableList<String> clientIds) {
static RdapAuthorization create(Role role, ImmutableSet<String> clientIds) {
return new AutoValue_RdapAuthorization(role, clientIds);
}
@ -54,9 +54,8 @@ public abstract class RdapAuthorization extends ImmutableObject {
}
public static final RdapAuthorization PUBLIC_AUTHORIZATION =
create(Role.PUBLIC, ImmutableList.of());
create(Role.PUBLIC, ImmutableSet.of());
public static final RdapAuthorization ADMINISTRATOR_AUTHORIZATION =
create(Role.ADMINISTRATOR, ImmutableList.of());
create(Role.ADMINISTRATOR, ImmutableSet.of());
}

4
java/google/registry/rdap/RdapDomainAction.java
View file

@ -29,7 +29,6 @@ import google.registry.request.Action;
import google.registry.request.HttpException.BadRequestException;
import google.registry.request.HttpException.NotFoundException;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import javax.inject.Inject;
import org.joda.time.DateTime;
@ -44,8 +43,7 @@ public class RdapDomainAction extends RdapActionBase {
public static final String PATH = "/rdap/domain/";
@Inject Clock clock;
@Inject RdapDomainAction() {}
@Inject public RdapDomainAction() {}
@Override
public String getHumanReadableObjectTypeName() {

4
java/google/registry/rdap/RdapDomainSearchAction.java
View file

@ -45,7 +45,6 @@ import google.registry.request.HttpException.NotFoundException;
import google.registry.request.HttpException.UnprocessableEntityException;
import google.registry.request.Parameter;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import google.registry.util.Idn;
import google.registry.util.NonFinalForTesting;
import java.net.InetAddress;
@ -83,11 +82,10 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
@Inject Clock clock;
@Inject @Parameter("name") Optional<String> nameParam;
@Inject @Parameter("nsLdhName") Optional<String> nsLdhNameParam;
@Inject @Parameter("nsIp") Optional<String> nsIpParam;
@Inject RdapDomainSearchAction() {}
@Inject public RdapDomainSearchAction() {}
@Override
public String getHumanReadableObjectTypeName() {

4
java/google/registry/rdap/RdapEntityAction.java
View file

@ -31,7 +31,6 @@ import google.registry.request.Action;
import google.registry.request.HttpException.BadRequestException;
import google.registry.request.HttpException.NotFoundException;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import java.util.Optional;
import javax.inject.Inject;
import org.joda.time.DateTime;
@ -58,8 +57,7 @@ public class RdapEntityAction extends RdapActionBase {
private static final Pattern ROID_PATTERN = Pattern.compile("[-_.a-zA-Z0-9]+");
@Inject Clock clock;
@Inject RdapEntityAction() {}
@Inject public RdapEntityAction() {}
@Override
public String getHumanReadableObjectTypeName() {

4
java/google/registry/rdap/RdapEntitySearchAction.java
View file

@ -39,7 +39,6 @@ import google.registry.request.HttpException.NotFoundException;
import google.registry.request.HttpException.UnprocessableEntityException;
import google.registry.request.Parameter;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
@ -84,11 +83,10 @@ public class RdapEntitySearchAction extends RdapSearchActionBase {
public static final String PATH = "/rdap/entities";
@Inject Clock clock;
@Inject @Parameter("fn") Optional<String> fnParam;
@Inject @Parameter("handle") Optional<String> handleParam;
@Inject @Parameter("subtype") Optional<String> subtypeParam;
@Inject RdapEntitySearchAction() {}
@Inject public RdapEntitySearchAction() {}
private enum QueryType {
FULL_NAME,

4
java/google/registry/rdap/RdapHelpAction.java
View file

@ -23,7 +23,6 @@ import google.registry.rdap.RdapJsonFormatter.BoilerplateType;
import google.registry.rdap.RdapMetrics.EndpointType;
import google.registry.request.Action;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import javax.inject.Inject;
/** RDAP (new WHOIS) action for help requests. */
@ -37,8 +36,7 @@ public class RdapHelpAction extends RdapActionBase {
public static final String PATH = "/rdap/help";
@Inject Clock clock;
@Inject RdapHelpAction() {}
@Inject public RdapHelpAction() {}
@Override
public String getHumanReadableObjectTypeName() {

3
java/google/registry/rdap/RdapJsonFormatter.java
View file

@ -29,7 +29,6 @@ import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import com.google.common.collect.Ordering;
import com.google.common.collect.Streams;
import com.google.common.flogger.FluentLogger;
import com.google.common.net.InetAddresses;
import com.googlecode.objectify.Key;
import google.registry.config.RdapNoticeDescriptor;
@ -83,8 +82,6 @@ public class RdapJsonFormatter {
@Inject @Config("rdapHelpMap") ImmutableMap<String, RdapNoticeDescriptor> rdapHelpMap;
@Inject RdapJsonFormatter() {}
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
/**
* What type of data to generate. Summary data includes only information about the object itself,
* while full data includes associated items (e.g. for domains, full data includes the hosts,

4
java/google/registry/rdap/RdapNameserverAction.java
View file

@ -29,7 +29,6 @@ import google.registry.request.Action;
import google.registry.request.HttpException.BadRequestException;
import google.registry.request.HttpException.NotFoundException;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import javax.inject.Inject;
import org.joda.time.DateTime;
@ -44,8 +43,7 @@ public class RdapNameserverAction extends RdapActionBase {
public static final String PATH = "/rdap/nameserver/";
@Inject Clock clock;
@Inject RdapNameserverAction() {}
@Inject public RdapNameserverAction() {}
@Override
public String getHumanReadableObjectTypeName() {

4
java/google/registry/rdap/RdapNameserverSearchAction.java
View file

@ -38,7 +38,6 @@ import google.registry.request.HttpException.NotFoundException;
import google.registry.request.HttpException.UnprocessableEntityException;
import google.registry.request.Parameter;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import google.registry.util.Idn;
import java.net.InetAddress;
import java.util.ArrayList;
@ -66,10 +65,9 @@ public class RdapNameserverSearchAction extends RdapSearchActionBase {
public static final String PATH = "/rdap/nameservers";
@Inject Clock clock;
@Inject @Parameter("name") Optional<String> nameParam;
@Inject @Parameter("ip") Optional<String> ipParam;
@Inject RdapNameserverSearchAction() {}
@Inject public RdapNameserverSearchAction() {}
@Override
public String getHumanReadableObjectTypeName() {

14
java/google/registry/rde/imports/RdeDomainImportAction.java
View file

@ -54,6 +54,7 @@ import google.registry.request.Action;
import google.registry.request.Parameter;
import google.registry.request.Response;
import google.registry.request.auth.Auth;
import google.registry.util.StringGenerator;
import google.registry.util.SystemClock;
import google.registry.xjc.JaxbFragment;
import google.registry.xjc.rdedomain.XjcRdeDomain;
@ -83,6 +84,7 @@ public class RdeDomainImportAction implements Runnable {
protected final String importBucketName;
protected final String importFileName;
protected final Optional<Integer> mapShards;
protected final StringGenerator stringGenerator;
@Inject
public RdeDomainImportAction(
@ -90,12 +92,14 @@ public class RdeDomainImportAction implements Runnable {
Response response,
@Config("rdeImportBucket") String importBucketName,
@Parameter(PATH) String importFileName,
@Parameter(PARAM_MAP_SHARDS) Optional<Integer> mapShards) {
@Parameter(PARAM_MAP_SHARDS) Optional<Integer> mapShards,
@Config("base64StringGenerator") StringGenerator stringGenerator) {
this.mrRunner = mrRunner;
this.response = response;
this.importBucketName = importBucketName;
this.importFileName = importFileName;
this.mapShards = mapShards;
this.stringGenerator = stringGenerator;
}
@Override
@ -122,7 +126,7 @@ public class RdeDomainImportAction implements Runnable {
* Creates a new {@link RdeDomainImportMapper}
*/
private RdeDomainImportMapper createMapper() {
return new RdeDomainImportMapper(importBucketName);
return new RdeDomainImportMapper(importBucketName, stringGenerator);
}
/** Mapper to import domains from an escrow file. */
@ -132,11 +136,13 @@ public class RdeDomainImportAction implements Runnable {
private static final long serialVersionUID = -7645091075256589374L;
private final String importBucketName;
private final StringGenerator stringGenerator;
private transient RdeImportUtils importUtils;
private transient DnsQueue dnsQueue;
public RdeDomainImportMapper(String importBucketName) {
public RdeDomainImportMapper(String importBucketName, StringGenerator stringGenerator) {
this.importBucketName = importBucketName;
this.stringGenerator = stringGenerator;
}
private RdeImportUtils getImportUtils() {
@ -196,7 +202,7 @@ public class RdeDomainImportAction implements Runnable {
createAutoRenewPollMessageForDomainImport(xjcDomain, historyEntry);
DomainResource domain =
XjcToDomainResourceConverter.convertDomain(
xjcDomain, autorenewBillingEvent, autorenewPollMessage);
xjcDomain, autorenewBillingEvent, autorenewPollMessage, stringGenerator);
getDnsQueue().addDomainRefreshTask(domain.getFullyQualifiedDomainName());
// Keep a list of "extra objects" that need to be saved along with the domain
// and add to it if necessary.

20
java/google/registry/rde/imports/XjcToDomainResourceConverter.java
View file

@ -42,8 +42,6 @@ import google.registry.model.registry.Registries;
import google.registry.model.registry.Registry;
import google.registry.model.transfer.TransferData;
import google.registry.model.transfer.TransferStatus;
import google.registry.util.NonFinalForTesting;
import google.registry.util.RandomStringGenerator;
import google.registry.util.StringGenerator;
import google.registry.util.XmlToEnumMapper;
import google.registry.xjc.domain.XjcDomainContactType;
@ -54,27 +52,12 @@ import google.registry.xjc.rdedomain.XjcRdeDomainElement;
import google.registry.xjc.rdedomain.XjcRdeDomainTransferDataType;
import google.registry.xjc.rgp.XjcRgpStatusType;
import google.registry.xjc.secdns.XjcSecdnsDsDataType;
import java.security.NoSuchAlgorithmException;
import java.security.ProviderException;
import java.security.SecureRandom;
import java.util.function.Function;
import org.joda.time.DateTime;
/** Utility class that converts an {@link XjcRdeDomainElement} into a {@link DomainResource}. */
final class XjcToDomainResourceConverter extends XjcToEppResourceConverter {
@NonFinalForTesting
static StringGenerator stringGenerator =
new RandomStringGenerator(StringGenerator.Alphabets.BASE_64, getRandom());
static SecureRandom getRandom() {
try {
return SecureRandom.getInstance("NativePRNG");
} catch (NoSuchAlgorithmException e) {
throw new ProviderException(e);
}
}
private static final XmlToEnumMapper<TransferStatus> TRANSFER_STATUS_MAPPER =
XmlToEnumMapper.create(TransferStatus.values());
@ -152,7 +135,8 @@ final class XjcToDomainResourceConverter extends XjcToEppResourceConverter {
static DomainResource convertDomain(
XjcRdeDomain domain,
BillingEvent.Recurring autoRenewBillingEvent,
PollMessage.Autorenew autoRenewPollMessage) {
PollMessage.Autorenew autoRenewPollMessage,
StringGenerator stringGenerator) {
GracePeriodConverter gracePeriodConverter =
new GracePeriodConverter(domain, Key.create(autoRenewBillingEvent));
DomainResource.Builder builder =

6
java/google/registry/reporting/icann/IcannReportingModule.java
View file

@ -25,7 +25,6 @@ import dagger.Provides;
import google.registry.bigquery.BigqueryConnection;
import google.registry.request.HttpException.BadRequestException;
import google.registry.request.Parameter;
import google.registry.util.SendEmailService;
import java.lang.annotation.Documented;
import java.lang.annotation.Retention;
import java.util.Optional;
@ -115,11 +114,6 @@ public final class IcannReportingModule {
}
}
@Provides
static SendEmailService provideSendEmailService() {
return new SendEmailService();
}
/** Dagger qualifier for the subdirectory we stage to/upload from. */
@Qualifier
@Documented

21
java/google/registry/reporting/spec11/Spec11EmailUtils.java
View file

@ -47,8 +47,10 @@ public class Spec11EmailUtils {
private final YearMonth yearMonth;
private final String alertSenderAddress;
private final String alertRecipientAddress;
private final String spec11ReplyToAddress;
private final String reportingBucket;
private final String spec11ReportDirectory;
private final String spec11EmailBodyTemplate;
private final GcsUtils gcsUtils;
private final Retrier retrier;
@ -58,6 +60,8 @@ public class Spec11EmailUtils {
YearMonth yearMonth,
@Config("alertSenderEmailAddress") String alertSenderAddress,
@Config("alertRecipientEmailAddress") String alertRecipientAddress,
@Config("spec11ReplyToEmailAddress") String spec11ReplyToAddress,
@Config("spec11EmailBodyTemplate") String spec11EmailBodyTemplate,
@Config("reportingBucket") String reportingBucket,
@Spec11ReportDirectory String spec11ReportDirectory,
GcsUtils gcsUtils,
@ -66,8 +70,10 @@ public class Spec11EmailUtils {
this.yearMonth = yearMonth;
this.alertSenderAddress = alertSenderAddress;
this.alertRecipientAddress = alertRecipientAddress;
this.spec11ReplyToAddress = spec11ReplyToAddress;
this.reportingBucket = reportingBucket;
this.spec11ReportDirectory = spec11ReportDirectory;
this.spec11EmailBodyTemplate = spec11EmailBodyTemplate;
this.gcsUtils = gcsUtils;
this.retrier = retrier;
}
@ -114,23 +120,24 @@ public class Spec11EmailUtils {
JSONObject reportJSON = new JSONObject(line);
String registrarEmail = reportJSON.getString(Spec11Pipeline.REGISTRAR_EMAIL_FIELD);
JSONArray threatMatches = reportJSON.getJSONArray(Spec11Pipeline.THREAT_MATCHES_FIELD);
StringBuilder body =
new StringBuilder("Hello registrar partner,\n")
.append("We have detected problems with the following domains:\n");
StringBuilder threatList = new StringBuilder();
for (int i = 0; i < threatMatches.length(); i++) {
ThreatMatch threatMatch = ThreatMatch.fromJSON(threatMatches.getJSONObject(i));
body.append(
threatList.append(
String.format(
"%s - %s\n", threatMatch.fullyQualifiedDomainName(), threatMatch.threatType()));
}
body.append("At the moment, no action is required. This is purely informatory.")
.append("Regards,\nGoogle Registry\n");
String body =
spec11EmailBodyTemplate
.replace("{REPLY_TO_EMAIL}", spec11ReplyToAddress)
.replace("{LIST_OF_THREATS}", threatList.toString());
Message msg = emailService.createMessage();
msg.setSubject(
String.format("Google Registry Monthly Threat Detector [%s]", yearMonth.toString()));
msg.setText(body.toString());
msg.setFrom(new InternetAddress(alertSenderAddress));
msg.setRecipient(RecipientType.TO, new InternetAddress(registrarEmail));
msg.addRecipient(RecipientType.TO, new InternetAddress(registrarEmail));
msg.addRecipient(RecipientType.BCC, new InternetAddress(spec11ReplyToAddress));
emailService.sendMessage(msg);
}

99
java/google/registry/repositories.bzl
View file

@ -72,6 +72,7 @@ def domain_registry_repositories(
omit_com_google_googlejavaformat_google_java_format = False,
omit_com_google_guava = False,
omit_com_google_guava_testlib = False,
omit_com_google_gwt_user = False,
omit_com_google_http_client = False,
omit_com_google_http_client_appengine = False,
omit_com_google_http_client_jackson2 = False,
@ -88,6 +89,7 @@ def domain_registry_repositories(
omit_com_google_template_soy = False,
omit_com_google_truth = False,
omit_com_google_truth_extensions_truth_java8_extension = False,
omit_com_googlecode_java_diff_utils_diffutils = False,
omit_com_googlecode_charts4j = False,
omit_com_googlecode_json_simple = False,
omit_com_ibm_icu_icu4j = False,
@ -117,6 +119,7 @@ def domain_registry_repositories(
omit_javax_inject = False,
omit_javax_mail = False,
omit_javax_servlet_api = False,
omit_javax_validation_api = False,
omit_javax_xml_bind_jaxb_api = False,
omit_javax_xml_soap_api = False,
omit_javax_xml_ws_jaxws_api = False,
@ -136,6 +139,8 @@ def domain_registry_repositories(
omit_org_apache_httpcomponents_httpcore = False,
omit_org_apache_mina_core = False,
omit_org_apache_sshd_core = False,
omit_org_apache_sshd_scp = False,
omit_org_apache_sshd_sftp = False,
omit_org_apache_tomcat_servlet_api = False,
omit_org_apache_tomcat_annotations_api = False,
omit_org_bouncycastle_bcpg_jdk15on = False,
@ -262,6 +267,8 @@ def domain_registry_repositories(
com_google_guava()
if not omit_com_google_guava_testlib:
com_google_guava_testlib()
if not omit_com_google_gwt_user:
com_google_gwt_user()
if not omit_com_google_http_client:
com_google_http_client()
if not omit_com_google_http_client_appengine:
@ -294,6 +301,8 @@ def domain_registry_repositories(
com_google_truth()
if not omit_com_google_truth_extensions_truth_java8_extension:
com_google_truth_extensions_truth_java8_extension()
if not omit_com_googlecode_java_diff_utils_diffutils:
com_googlecode_java_diff_utils_diffutils()
if not omit_com_googlecode_charts4j:
com_googlecode_charts4j()
if not omit_com_googlecode_json_simple:
@ -352,6 +361,8 @@ def domain_registry_repositories(
javax_mail()
if not omit_javax_servlet_api:
javax_servlet_api()
if not omit_javax_validation_api:
javax_validation_api()
if not omit_javax_xml_bind_jaxb_api:
javax_xml_bind_jaxb_api()
if not omit_javax_xml_soap_api:
@ -388,6 +399,10 @@ def domain_registry_repositories(
org_apache_mina_core()
if not omit_org_apache_sshd_core:
org_apache_sshd_core()
if not omit_org_apache_sshd_scp:
org_apache_sshd_scp()
if not omit_org_apache_sshd_sftp:
org_apache_sshd_sftp()
if not omit_org_apache_tomcat_servlet_api:
org_apache_tomcat_servlet_api()
if not omit_org_apache_tomcat_annotations_api:
@ -1342,6 +1357,23 @@ def com_google_guava_testlib():
],
)
def com_google_gwt_user():
java_import_external(
name = "com_google_gwt_user",
neverlink = 1,
licenses = ["notice"], # GWT Terms
jar_sha256 = "9f420f0d0c2f177d71cb1794b3be1418f9755f6e4181101af3951b8302b9556d",
jar_urls = [
"http://maven.ibiblio.org/maven2/com/google/gwt/gwt-user/2.8.2/gwt-user-2.8.2.jar",
"http://repo1.maven.org/maven2/com/google/gwt/gwt-user/2.8.2/gwt-user-2.8.2.jar",
],
deps = [
"@javax_validation_api",
"@javax_servlet_api",
"@org_w3c_css_sac",
],
)
def com_google_http_client():
java_import_external(
name = "com_google_http_client",
@ -1547,6 +1579,7 @@ def com_google_truth():
"@junit",
"@com_google_auto_value",
"@com_google_errorprone_error_prone_annotations",
"@com_googlecode_java_diff_utils_diffutils",
],
)
@ -1577,6 +1610,17 @@ def com_googlecode_charts4j():
licenses = ["notice"], # The MIT License
)
def com_googlecode_java_diff_utils_diffutils():
java_import_external(
name = "com_googlecode_java_diff_utils_diffutils",
licenses = ["notice"], # The Apache Software License, Version 2.0
jar_sha256 = "61ba4dc49adca95243beaa0569adc2a23aedb5292ae78aa01186fa782ebdc5c2",
jar_urls = [
"http://maven.ibiblio.org/maven2/com/googlecode/java-diff-utils/diffutils/1.3.0/diffutils-1.3.0.jar",
"http://repo1.maven.org/maven2/com/googlecode/java-diff-utils/diffutils/1.3.0/diffutils-1.3.0.jar",
],
)
def com_googlecode_json_simple():
java_import_external(
name = "com_googlecode_json_simple",
@ -1907,6 +1951,17 @@ def javax_servlet_api():
licenses = ["notice"], # Apache
)
def javax_validation_api():
java_import_external(
name = "javax_validation_api",
licenses = ["notice"], # Apache License, Version 2.0
jar_sha256 = "e459f313ebc6db2483f8ceaad39af07086361b474fa92e40f442e8de5d9895dc",
jar_urls = [
"http://maven.ibiblio.org/maven2/javax/validation/validation-api/1.0.0.GA/validation-api-1.0.0.GA.jar",
"http://repo1.maven.org/maven2/javax/validation/validation-api/1.0.0.GA/validation-api-1.0.0.GA.jar",
],
)
def javax_xml_bind_jaxb_api():
java_import_external(
name = "javax_xml_bind_jaxb_api",
@ -2195,15 +2250,51 @@ def org_apache_mina_core():
def org_apache_sshd_core():
java_import_external(
name = "org_apache_sshd_core",
jar_sha256 = "5630fa11f7e2f7f5b6b7e6b9be06e476715dfb48db37998b4b7c3eea098d86ff",
# Apache 2.0 License
# http://www.apache.org/licenses/LICENSE-2.0
# Apache License, Version 2.0
# http://www.apache.org/licenses/LICENSE-2.0.txt
licenses = ["notice"],
jar_sha256 = "00c944fac00dec2e7ace4052e0a52c772ca3fa2653918bbcfadf7100df022e25",
jar_urls = [
"http://maven.ibiblio.org/maven2/org/apache/sshd/sshd-core/1.2.0/sshd-core-1.2.0.jar",
"http://repo1.maven.org/maven2/org/apache/sshd/sshd-core/1.2.0/sshd-core-1.2.0.jar",
"http://repo1.maven.org/maven2/org/apache/sshd/sshd-core/2.0.0/sshd-core-2.0.0.jar",
"http://maven.ibiblio.org/maven2/org/apache/sshd/sshd-core/2.0.0/sshd-core-2.0.0.jar",
],
licenses = ["notice"], # Apache 2.0 License
deps = ["@org_slf4j_api"],
)
def org_apache_sshd_scp():
java_import_external(
name = "org_apache_sshd_scp",
# Apache 2.0 License
# http://www.apache.org/licenses/LICENSE-2.0
# Apache License, Version 2.0
# http://www.apache.org/licenses/LICENSE-2.0.txt
licenses = ["notice"],
jar_sha256 = "ae32fcc16ab0a0ae04655b8832676b41199814184dc50028b3c6aa61053635ca",
jar_urls = [
"http://repo1.maven.org/maven2/org/apache/sshd/sshd-scp/2.0.0/sshd-scp-2.0.0.jar",
"http://maven.ibiblio.org/maven2/org/apache/sshd/sshd-scp/2.0.0/sshd-scp-2.0.0.jar",
],
deps = ["@org_apache_sshd_core"],
)
def org_apache_sshd_sftp():
java_import_external(
name = "org_apache_sshd_sftp",
# Apache 2.0 License
# http://www.apache.org/licenses/LICENSE-2.0
# Apache License, Version 2.0
# http://www.apache.org/licenses/LICENSE-2.0.txt
licenses = ["notice"],
jar_sha256 = "0504af9a4afcaf61be9f0b56d3cfc76a9187a654e297bc57b7fa81aa76bb8cb0",
jar_urls = [
"http://repo1.maven.org/maven2/org/apache/sshd/sshd-sftp/2.0.0/sshd-sftp-2.0.0.jar",
"http://maven.ibiblio.org/maven2/org/apache/sshd/sshd-sftp/2.0.0/sshd-sftp-2.0.0.jar",
],
deps = ["@org_apache_sshd_core"],
)
def org_apache_tomcat_servlet_api():
java_import_external(
name = "org_apache_tomcat_servlet_api",

95
java/google/registry/request/Modules.java
View file

@ -15,13 +15,9 @@
package google.registry.request;
import static com.google.appengine.api.datastore.DatastoreServiceFactory.getDatastoreService;
import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.api.client.extensions.appengine.http.UrlFetchTransport;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.extensions.appengine.auth.oauth2.AppIdentityCredential;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
@ -31,15 +27,8 @@ import com.google.appengine.api.urlfetch.URLFetchService;
import com.google.appengine.api.urlfetch.URLFetchServiceFactory;
import com.google.appengine.api.users.UserService;
import com.google.appengine.api.users.UserServiceFactory;
import dagger.Binds;
import dagger.Module;
import dagger.Provides;
import google.registry.keyring.api.KeyModule.Key;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.Set;
import java.util.function.Function;
import javax.inject.Provider;
import javax.inject.Singleton;
/** Dagger modules for App Engine services and other vendor classes. */
@ -120,88 +109,4 @@ public final class Modules {
}
}
}
/**
* Dagger module providing {@link AppIdentityCredential}.
*
* <p>This can be used to authenticate to Google APIs using the identity of your GAE app.
*
* @see UseAppIdentityCredentialForGoogleApisModule
*/
@Module
public static final class AppIdentityCredentialModule {
@Provides
static Function<Set<String>, AppIdentityCredential> provideAppIdentityCredential() {
return AppIdentityCredential::new;
}
}
/**
* Dagger module causing Google APIs requests to be authorized with your GAE app identity.
*
* <p>You must also use the {@link AppIdentityCredentialModule}.
*/
@Module
public abstract static class UseAppIdentityCredentialForGoogleApisModule {
@Binds
abstract Function<Set<String>, ? extends HttpRequestInitializer> provideHttpRequestInitializer(
Function<Set<String>, AppIdentityCredential> credential);
}
/**
* Module indicating Google API requests should be authorized with JSON {@link GoogleCredential}.
*
* <p>This is useful when configuring a component that runs the registry outside of the App Engine
* environment, for example, in a command line environment.
*
* <p>You must also use the {@link GoogleCredentialModule}.
*/
@Module
public abstract static class UseGoogleCredentialForGoogleApisModule {
@Binds
abstract Function<Set<String>, ? extends HttpRequestInitializer> provideHttpRequestInitializer(
Function<Set<String>, GoogleCredential> credential);
}
/**
* Dagger module providing {@link GoogleCredential} from a JSON key file contents.
*
* <p>This satisfies the {@link HttpRequestInitializer} interface for authenticating Google APIs
* requests, just like {@link AppIdentityCredential}.
*
* <p>But we consider GAE authentication more desirable and easier to manage operations-wise. So
* this authentication method should only be used for the following situations:
*
* <ol>
* <li>Locally-running programs (which aren't executing on the App Engine platform)
* <li>Spreadsheet service (which can't use {@link AppIdentityCredential} due to an old library)
* </ol>
*
* @see google.registry.keyring.api.Keyring#getJsonCredential()
*/
@Module
public static final class GoogleCredentialModule {
@Provides
@Singleton
static GoogleCredential provideGoogleCredential(
NetHttpTransport netHttpTransport,
JsonFactory jsonFactory,
@Key("jsonCredential") String jsonCredential) {
try {
return GoogleCredential.fromStream(
new ByteArrayInputStream(jsonCredential.getBytes(UTF_8)),
netHttpTransport,
jsonFactory);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
@Provides
static Function<Set<String>, GoogleCredential> provideScopedGoogleCredential(
final Provider<GoogleCredential> googleCredentialProvider) {
return scopes -> googleCredentialProvider.get().createScoped(scopes);
}
}
}

10
java/google/registry/request/auth/AuthResult.java
View file

@ -36,6 +36,16 @@ public abstract class AuthResult {
return authLevel() != AuthLevel.NONE;
}
public String userIdForLogging() {
return userAuthInfo()
.map(
userAuthInfo ->
String.format(
"%s %s",
userAuthInfo.isUserAdmin() ? "admin" : "user", userAuthInfo.user().getEmail()))
.orElse("<logged-out user>");
}
public static AuthResult create(AuthLevel authLevel) {
return new AutoValue_AuthResult(authLevel, Optional.empty());
}

25
java/google/registry/tldconfig/idn/IdnLabelValidator.java
View file

@ -20,7 +20,6 @@ import static google.registry.tldconfig.idn.IdnTableEnum.JA;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import google.registry.util.Idn;
import google.registry.util.NonFinalForTesting;
import java.util.Optional;
/** Validates whether a given IDN label can be provisioned for a particular TLD. */
@ -30,10 +29,20 @@ public final class IdnLabelValidator {
private static final ImmutableList<IdnTableEnum> DEFAULT_IDN_TABLES =
ImmutableList.of(EXTENDED_LATIN, JA);
private static final ImmutableMap<String, ImmutableList<IdnTableEnum>>
DEFAULT_IDN_TABLE_LISTS_PER_TLD =
ImmutableMap.of("xn--q9jyb4c", ImmutableList.of(EXTENDED_LATIN, JA));
/** Some TLDs have their own IDN tables, configured here. */
@NonFinalForTesting
private static ImmutableMap<String, ImmutableList<IdnTableEnum>> idnTableListsPerTld =
ImmutableMap.of("xn--q9jyb4c", ImmutableList.of(EXTENDED_LATIN, JA));
private ImmutableMap<String, ImmutableList<IdnTableEnum>> idnTableListsPerTld;
IdnLabelValidator(ImmutableMap<String, ImmutableList<IdnTableEnum>> indTableListsPerTld) {
this.idnTableListsPerTld = indTableListsPerTld;
}
public static IdnLabelValidator createDefaultIdnLabelValidator() {
return new IdnLabelValidator(DEFAULT_IDN_TABLE_LISTS_PER_TLD);
}
/**
* Returns name of first matching {@link IdnTable} if domain label is valid for the given TLD.
@ -41,16 +50,14 @@ public final class IdnLabelValidator {
* <p>A label is valid if it is considered valid by at least one configured IDN table for that
* TLD. If no match is found, an absent value is returned.
*/
public static Optional<String> findValidIdnTableForTld(String label, String tld) {
public Optional<String> findValidIdnTableForTld(String label, String tld) {
String unicodeString = Idn.toUnicode(label);
for (IdnTableEnum idnTable
: Optional.ofNullable(idnTableListsPerTld.get(tld)).orElse(DEFAULT_IDN_TABLES)) {
for (IdnTableEnum idnTable :
Optional.ofNullable(idnTableListsPerTld.get(tld)).orElse(DEFAULT_IDN_TABLES)) {
if (idnTable.getTable().isValidLabel(unicodeString)) {
return Optional.of(idnTable.getTable().getName());
}
}
return Optional.empty();
}
private IdnLabelValidator() {}
}

63
java/google/registry/tmch/LordnTask.java → java/google/registry/tmch/LordnTaskUtils.java
View file

@ -15,36 +15,22 @@
package google.registry.tmch;
import static com.google.appengine.api.taskqueue.QueueFactory.getQueue;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Preconditions.checkState;
import static google.registry.model.ofy.ObjectifyService.ofy;
import com.google.appengine.api.taskqueue.LeaseOptions;
import com.google.appengine.api.taskqueue.Queue;
import com.google.appengine.api.taskqueue.TaskHandle;
import com.google.appengine.api.taskqueue.TaskOptions;
import com.google.appengine.api.taskqueue.TaskOptions.Method;
import com.google.appengine.api.taskqueue.TransientFailureException;
import com.google.apphosting.api.DeadlineExceededException;
import com.google.common.base.Joiner;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import com.google.common.util.concurrent.Uninterruptibles;
import google.registry.model.domain.DomainResource;
import google.registry.model.registrar.Registrar;
import google.registry.util.NonFinalForTesting;
import google.registry.util.TaskQueueUtils;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import org.joda.time.DateTime;
import org.joda.time.Duration;
/**
* Helper methods for creating tasks containing CSV line data in the lordn-sunrise and lordn-claims
* queues based on DomainResource changes.
*/
public class LordnTask {
public final class LordnTaskUtils {
public static final String QUEUE_SUNRISE = "lordn-sunrise";
public static final String QUEUE_CLAIMS = "lordn-claims";
@ -52,51 +38,6 @@ public class LordnTask {
+ "registration-datetime,ack-datetime,application-datetime";
public static final String COLUMNS_SUNRISE = "roid,domain-name,SMD-id,registrar-id,"
+ "registration-datetime,application-datetime";
private static final Duration LEASE_PERIOD = Duration.standardHours(1);
@NonFinalForTesting
private static Long backOffMillis = 2000L;
/**
* Converts a list of queue tasks, each containing a row of CSV data, into a single newline-
* delimited String.
*/
public static String convertTasksToCsv(List<TaskHandle> tasks, DateTime now, String columns) {
String header = String.format("1,%s,%d\n%s\n", now, tasks.size(), columns);
StringBuilder csv = new StringBuilder(header);
for (TaskHandle task : checkNotNull(tasks)) {
String payload = new String(task.getPayload());
if (!Strings.isNullOrEmpty(payload)) {
csv.append(payload).append("\n");
}
}
return csv.toString();
}
/** Leases and returns all tasks from the queue with the specified tag tld, in batches. */
public static List<TaskHandle> loadAllTasks(Queue queue, String tld) {
ImmutableList.Builder<TaskHandle> allTasks = new ImmutableList.Builder<>();
int numErrors = 0;
long backOff = backOffMillis;
while (true) {
try {
List<TaskHandle> tasks = queue.leaseTasks(LeaseOptions.Builder
.withTag(tld)
.leasePeriod(LEASE_PERIOD.getMillis(), TimeUnit.MILLISECONDS)
.countLimit(TaskQueueUtils.getBatchSize()));
allTasks.addAll(tasks);
if (tasks.isEmpty()) {
return allTasks.build();
}
} catch (TransientFailureException | DeadlineExceededException e) {
if (++numErrors >= 3) {
throw new RuntimeException("Error leasing tasks", e);
}
Uninterruptibles.sleepUninterruptibly(backOff, TimeUnit.MILLISECONDS);
backOff *= 2;
}
}
}
/**
* Enqueues a task in the LORDN queue representing a line of CSV for LORDN export.
@ -156,4 +97,6 @@ public class LordnTask {
// have null iana ids.
return String.valueOf(registrar.get().getIanaIdentifier());
}
private LordnTaskUtils() {}
}

90
java/google/registry/tmch/NordnUploadAction.java
View file

@ -19,21 +19,27 @@ import static com.google.appengine.api.taskqueue.TaskOptions.Builder.withUrl;
import static com.google.appengine.api.urlfetch.FetchOptions.Builder.validateCertificate;
import static com.google.appengine.api.urlfetch.HTTPMethod.POST;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.net.HttpHeaders.LOCATION;
import static com.google.common.net.MediaType.CSV_UTF_8;
import static google.registry.tmch.LordnTask.COLUMNS_CLAIMS;
import static google.registry.tmch.LordnTask.COLUMNS_SUNRISE;
import static google.registry.tmch.LordnTask.convertTasksToCsv;
import static google.registry.tmch.LordnTaskUtils.COLUMNS_CLAIMS;
import static google.registry.tmch.LordnTaskUtils.COLUMNS_SUNRISE;
import static google.registry.util.UrlFetchUtils.getHeaderFirst;
import static google.registry.util.UrlFetchUtils.setPayloadMultipart;
import static java.nio.charset.StandardCharsets.UTF_8;
import static javax.servlet.http.HttpServletResponse.SC_ACCEPTED;
import com.google.appengine.api.taskqueue.LeaseOptions;
import com.google.appengine.api.taskqueue.Queue;
import com.google.appengine.api.taskqueue.TaskHandle;
import com.google.appengine.api.taskqueue.TaskOptions;
import com.google.appengine.api.taskqueue.TransientFailureException;
import com.google.appengine.api.urlfetch.HTTPRequest;
import com.google.appengine.api.urlfetch.HTTPResponse;
import com.google.appengine.api.urlfetch.URLFetchService;
import com.google.apphosting.api.DeadlineExceededException;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import com.google.common.flogger.FluentLogger;
import google.registry.config.RegistryConfig.Config;
import google.registry.request.Action;
@ -41,6 +47,7 @@ import google.registry.request.Parameter;
import google.registry.request.RequestParameters;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import google.registry.util.Retrier;
import google.registry.util.TaskQueueUtils;
import google.registry.util.UrlFetchException;
import java.io.IOException;
@ -48,29 +55,30 @@ import java.net.URL;
import java.util.List;
import java.util.Optional;
import java.util.Random;
import java.util.concurrent.TimeUnit;
import javax.inject.Inject;
import org.joda.time.DateTime;
import org.joda.time.Duration;
/**
* Action that reads the NORDN pull queues, uploads claims and sunrise marks data to TMCH, and
* enqueues subsequent upload verification tasks. A unique actionLogId is generated and passed
* along to the verify action so that connected verify tasks can be identified by looking at logs.
* enqueues subsequent upload verification tasks. A unique actionLogId is generated and passed along
* to the verify action so that connected verify tasks can be identified by looking at logs.
*
* @see NordnVerifyAction
*/
@Action(
path = NordnUploadAction.PATH,
method = Action.Method.POST,
automaticallyPrintOk = true,
auth = Auth.AUTH_INTERNAL_ONLY
)
path = NordnUploadAction.PATH,
method = Action.Method.POST,
automaticallyPrintOk = true,
auth = Auth.AUTH_INTERNAL_ONLY)
public final class NordnUploadAction implements Runnable {
static final String PATH = "/_dr/task/nordnUpload";
static final String LORDN_PHASE_PARAM = "lordn-phase";
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private static final Duration LEASE_PERIOD = Duration.standardHours(1);
/**
* A unique (enough) id that is outputted in log lines to make it clear which log lines are
@ -80,6 +88,8 @@ public final class NordnUploadAction implements Runnable {
private final String actionLogId = String.valueOf(1000000000 + new Random().nextInt(1000000000));
@Inject Clock clock;
@Inject Retrier retrier;
@Inject @Config("insecureRandom") Random random;
@Inject LordnRequestInitializer lordnRequestInitializer;
@Inject URLFetchService fetchService;
@Inject @Config("tmchMarksdbUrl") String tmchMarksdbUrl;
@ -107,15 +117,54 @@ public final class NordnUploadAction implements Runnable {
}
}
/**
* Converts a list of queue tasks, each containing a row of CSV data, into a single newline-
* delimited String.
*/
static String convertTasksToCsv(List<TaskHandle> tasks, DateTime now, String columns) {
String header = String.format("1,%s,%d\n%s\n", now, tasks.size(), columns);
StringBuilder csv = new StringBuilder(header);
for (TaskHandle task : checkNotNull(tasks)) {
String payload = new String(task.getPayload(), UTF_8);
if (!Strings.isNullOrEmpty(payload)) {
csv.append(payload).append("\n");
}
}
return csv.toString();
}
/** Leases and returns all tasks from the queue with the specified tag tld, in batches. */
List<TaskHandle> loadAllTasks(Queue queue, String tld) {
ImmutableList.Builder<TaskHandle> allTasks = new ImmutableList.Builder<>();
while (true) {
List<TaskHandle> tasks =
retrier.callWithRetry(
() ->
queue.leaseTasks(
LeaseOptions.Builder.withTag(tld)
.leasePeriod(LEASE_PERIOD.getMillis(), TimeUnit.MILLISECONDS)
.countLimit(TaskQueueUtils.getBatchSize())),
TransientFailureException.class,
DeadlineExceededException.class);
if (tasks.isEmpty()) {
return allTasks.build();
}
allTasks.addAll(tasks);
}
}
private void processLordnTasks() throws IOException {
checkArgument(phase.equals(PARAM_LORDN_PHASE_SUNRISE)
|| phase.equals(PARAM_LORDN_PHASE_CLAIMS),
"Invalid phase specified to Nordn servlet: %s.", phase);
DateTime now = clock.nowUtc();
Queue queue = getQueue(
phase.equals(PARAM_LORDN_PHASE_SUNRISE) ? LordnTask.QUEUE_SUNRISE : LordnTask.QUEUE_CLAIMS);
Queue queue =
getQueue(
phase.equals(PARAM_LORDN_PHASE_SUNRISE)
? LordnTaskUtils.QUEUE_SUNRISE
: LordnTaskUtils.QUEUE_CLAIMS);
String columns = phase.equals(PARAM_LORDN_PHASE_SUNRISE) ? COLUMNS_SUNRISE : COLUMNS_CLAIMS;
List<TaskHandle> tasks = LordnTask.loadAllTasks(queue, tld);
List<TaskHandle> tasks = loadAllTasks(queue, tld);
if (!tasks.isEmpty()) {
String csvData = convertTasksToCsv(tasks, now, columns);
uploadCsvToLordn(String.format("/LORDN/%s/%s", tld, phase), csvData);
@ -138,23 +187,26 @@ public final class NordnUploadAction implements Runnable {
"LORDN upload task %s: Sending to URL: %s ; data: %s", actionLogId, url, csvData);
HTTPRequest req = new HTTPRequest(new URL(url), POST, validateCertificate().setDeadline(60d));
lordnRequestInitializer.initialize(req, tld);
setPayloadMultipart(req, "file", "claims.csv", CSV_UTF_8, csvData);
setPayloadMultipart(req, "file", "claims.csv", CSV_UTF_8, csvData, random);
HTTPResponse rsp = fetchService.fetch(req);
logger.atInfo().log(
"LORDN upload task %s response: HTTP response code %d, response data: %s",
actionLogId, rsp.getResponseCode(), rsp.getContent());
if (rsp.getResponseCode() != SC_ACCEPTED) {
throw new UrlFetchException(
String.format("LORDN upload task %s error: Failed to upload LORDN claims to MarksDB",
actionLogId),
req, rsp);
String.format(
"LORDN upload task %s error: Failed to upload LORDN claims to MarksDB", actionLogId),
req,
rsp);
}
Optional<String> location = getHeaderFirst(rsp, LOCATION);
if (!location.isPresent()) {
throw new UrlFetchException(
String.format("LORDN upload task %s error: MarksDB failed to provide a Location header",
String.format(
"LORDN upload task %s error: MarksDB failed to provide a Location header",
actionLogId),
req, rsp);
req,
rsp);
}
getQueue(NordnVerifyAction.QUEUE).add(makeVerifyTask(new URL(location.get())));
}

84
java/google/registry/tmch/TmchCertificateAuthority.java
View file

@ -15,7 +15,7 @@
package google.registry.tmch;
import static google.registry.config.RegistryConfig.ConfigModule.TmchCaMode.PILOT;
import static google.registry.config.RegistryConfig.getSingletonCachePersistDuration;
import static google.registry.config.RegistryConfig.ConfigModule.TmchCaMode.PRODUCTION;
import static google.registry.config.RegistryConfig.getSingletonCacheRefreshDuration;
import static google.registry.util.ResourceUtils.readResourceUtf8;
import static java.util.concurrent.TimeUnit.MILLISECONDS;
@ -23,17 +23,16 @@ import static java.util.concurrent.TimeUnit.MILLISECONDS;
import com.google.common.cache.CacheBuilder;
import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
import com.google.common.collect.ImmutableMap;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryConfig.ConfigModule.TmchCaMode;
import google.registry.model.tmch.TmchCrl;
import google.registry.util.Clock;
import google.registry.util.NonFinalForTesting;
import google.registry.util.SystemClock;
import google.registry.util.X509Utils;
import java.security.GeneralSecurityException;
import java.security.cert.CertificateParsingException;
import java.security.cert.X509CRL;
import java.security.cert.X509Certificate;
import java.util.concurrent.ExecutionException;
import javax.annotation.concurrent.Immutable;
import javax.annotation.concurrent.ThreadSafe;
import javax.inject.Inject;
@ -58,9 +57,12 @@ public final class TmchCertificateAuthority {
private static final String CRL_PILOT_FILE = "icann-tmch-pilot.crl";
private final TmchCaMode tmchCaMode;
private final Clock clock;
public @Inject TmchCertificateAuthority(@Config("tmchCaMode") TmchCaMode tmchCaMode) {
@Inject
public TmchCertificateAuthority(@Config("tmchCaMode") TmchCaMode tmchCaMode, Clock clock) {
this.tmchCaMode = tmchCaMode;
this.clock = clock;
}
/**
@ -70,8 +72,8 @@ public final class TmchCertificateAuthority {
* string into an X509CRL instance is expensive and should itself be cached.
*
* <p>Note that the stored CRL won't exist for tests, and on deployed environments will always
* correspond to the correct CRL for the given TMCH CA mode because {@link TmchCrlAction} can
* only persist the correct one for this given environment.
* correspond to the correct CRL for the given TMCH CA mode because {@link TmchCrlAction} can only
* persist the correct one for this given environment.
*/
private static final LoadingCache<TmchCaMode, X509CRL> CRL_CACHE =
CacheBuilder.newBuilder()
@ -89,37 +91,28 @@ public final class TmchCertificateAuthority {
crlContents = storedCrl.getCrl();
}
X509CRL crl = X509Utils.loadCrl(crlContents);
try {
crl.verify(ROOT_CACHE.get(tmchCaMode).getPublicKey());
return crl;
} catch (ExecutionException e) {
if (e.getCause() instanceof GeneralSecurityException) {
throw (GeneralSecurityException) e.getCause();
} else {
throw new RuntimeException("Unexpected exception while loading CRL", e);
}
}
}});
crl.verify(ROOT_CERTS.get(tmchCaMode).getPublicKey());
return crl;
}
});
/** A cached function that loads the CRT from a jar resource. */
private static final LoadingCache<TmchCaMode, X509Certificate> ROOT_CACHE =
CacheBuilder.newBuilder()
.expireAfterWrite(getSingletonCachePersistDuration().getMillis(), MILLISECONDS)
.build(
new CacheLoader<TmchCaMode, X509Certificate>() {
@Override
public X509Certificate load(final TmchCaMode tmchCaMode)
throws GeneralSecurityException {
String file = (tmchCaMode == PILOT) ? ROOT_CRT_PILOT_FILE : ROOT_CRT_FILE;
X509Certificate root =
X509Utils.loadCertificate(
readResourceUtf8(TmchCertificateAuthority.class, file));
root.checkValidity(clock.nowUtc().toDate());
return root;
}});
/** CRTs from a jar resource. */
private static final ImmutableMap<TmchCaMode, X509Certificate> ROOT_CERTS =
loadRootCertificates();
@NonFinalForTesting
private static Clock clock = new SystemClock();
private static ImmutableMap<TmchCaMode, X509Certificate> loadRootCertificates() {
try {
return ImmutableMap.of(
PILOT,
X509Utils.loadCertificate(
readResourceUtf8(TmchCertificateAuthority.class, ROOT_CRT_PILOT_FILE)),
PRODUCTION,
X509Utils.loadCertificate(
readResourceUtf8(TmchCertificateAuthority.class, ROOT_CRT_FILE)));
} catch (CertificateParsingException e) {
throw new RuntimeException(e);
}
}
/**
* Check that {@code cert} is signed by the ICANN TMCH CA root and not revoked.
@ -132,7 +125,7 @@ public final class TmchCertificateAuthority {
*/
public void verify(X509Certificate cert) throws GeneralSecurityException {
synchronized (TmchCertificateAuthority.class) {
X509Utils.verifyCertificate(getRoot(), getCrl(), cert, clock.nowUtc().toDate());
X509Utils.verifyCertificate(getAndValidateRoot(), getCrl(), cert, clock.nowUtc().toDate());
}
}
@ -145,21 +138,26 @@ public final class TmchCertificateAuthority {
* refreshes itself.
*
* @throws GeneralSecurityException for unsupported protocols, certs not signed by the TMCH,
* incorrect keys, and for invalid, old, not-yet-valid or revoked certificates.
* incorrect keys, and for invalid, old, not-yet-valid or revoked certificates.
* @see X509Utils#verifyCrl
*/
public void updateCrl(String asciiCrl, String url) throws GeneralSecurityException {
X509CRL crl = X509Utils.loadCrl(asciiCrl);
X509Utils.verifyCrl(getRoot(), getCrl(), crl, clock.nowUtc().toDate());
X509Utils.verifyCrl(getAndValidateRoot(), getCrl(), crl, clock.nowUtc().toDate());
TmchCrl.set(asciiCrl, url);
}
public X509Certificate getRoot() throws GeneralSecurityException {
public X509Certificate getAndValidateRoot() throws GeneralSecurityException {
try {
return ROOT_CACHE.get(tmchCaMode);
X509Certificate root = ROOT_CERTS.get(tmchCaMode);
// The current production certificate expires on 2023-07-23. Future code monkey be reminded,
// if you are looking at this code because the next line throws an exception, ask ICANN for a
// new root certificate! (preferably before the current one expires...)
root.checkValidity(clock.nowUtc().toDate());
return root;
} catch (Exception e) {
if (e.getCause() instanceof GeneralSecurityException) {
throw (GeneralSecurityException) e.getCause();
if (e instanceof GeneralSecurityException) {
throw (GeneralSecurityException) e;
} else if (e instanceof RuntimeException) {
throw (RuntimeException) e;
}

89
java/google/registry/tools/AppEngineConnection.java
View file

@ -15,7 +15,6 @@
package google.registry.tools;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Suppliers.memoize;
import static com.google.common.net.HttpHeaders.X_REQUESTED_WITH;
import static com.google.common.net.MediaType.JSON_UTF_8;
import static google.registry.security.JsonHttp.JSON_SAFETY_PREFIX;
@ -27,48 +26,55 @@ import com.google.api.client.http.HttpHeaders;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestFactory;
import com.google.api.client.http.HttpResponse;
import com.google.common.base.Supplier;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.io.CharStreams;
import com.google.common.net.HostAndPort;
import com.google.common.net.MediaType;
import com.google.re2j.Matcher;
import com.google.re2j.Pattern;
import google.registry.security.XsrfTokenManager;
import google.registry.tools.CommandWithConnection.Connection;
import google.registry.config.RegistryConfig;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.Map;
import javax.annotation.Nullable;
import javax.inject.Inject;
import org.json.simple.JSONValue;
/** An http connection to the appengine server. */
class AppEngineConnection implements Connection {
/**
* An http connection to an appengine server.
*
* <p>By default - connects to the TOOLS service. To create a Connection to another service, call
* the {@link #withService} function.
*/
class AppEngineConnection {
/** Pattern to heuristically extract title tag contents in HTML responses. */
private static final Pattern HTML_TITLE_TAG_PATTERN = Pattern.compile("<title>(.*?)</title>");
@Inject HttpRequestFactory requestFactory;
@Inject AppEngineConnectionFlags flags;
@Inject XsrfTokenManager xsrfTokenManager;
private final Service service;
@Inject
AppEngineConnection() {}
AppEngineConnection() {
service = Service.TOOLS;
}
/**
* Memoized XSRF security token.
*
* <p>Computing this is expensive since it needs to load {@code ServerSecret} so do it once.
*/
private final Supplier<String> xsrfToken =
memoize(() -> xsrfTokenManager.generateToken(getUserId()));
private AppEngineConnection(Service service, HttpRequestFactory requestFactory) {
this.service = service;
this.requestFactory = requestFactory;
}
@Override
public void prefetchXsrfToken() {
// Cause XSRF token to be fetched, and then stay resident in cache (since it's memoized).
xsrfToken.get();
enum Service {
DEFAULT,
TOOLS,
BACKEND,
PUBAPI
}
/** Returns a copy of this connection that talks to a different service. */
public AppEngineConnection withService(Service service) {
return new AppEngineConnection(service, requestFactory);
}
/** Returns the contents of the title tag in the given HTML, or null if not found. */
@ -85,7 +91,8 @@ class AppEngineConnection implements Connection {
private String internalSend(
String endpoint, Map<String, ?> params, MediaType contentType, @Nullable byte[] payload)
throws IOException {
GenericUrl url = new GenericUrl(String.format("%s%s", getServerUrl(), endpoint));
GenericUrl url = new GenericUrl(getServer());
url.setRawPath(endpoint);
url.putAll(params);
HttpRequest request =
(payload != null)
@ -120,23 +127,20 @@ class AppEngineConnection implements Connection {
}
}
// TODO(b/111123862): Rename this to sendPostRequest()
@Override
public String send(String endpoint, Map<String, ?> params, MediaType contentType, byte[] payload)
public String sendPostRequest(
String endpoint, Map<String, ?> params, MediaType contentType, byte[] payload)
throws IOException {
return internalSend(endpoint, params, contentType, checkNotNull(payload, "payload"));
}
@Override
public String sendGetRequest(String endpoint, Map<String, ?> params) throws IOException {
return internalSend(endpoint, params, MediaType.PLAIN_TEXT_UTF_8, null);
}
@Override
@SuppressWarnings("unchecked")
public Map<String, Object> sendJson(String endpoint, Map<String, ?> object) throws IOException {
String response =
send(
sendPostRequest(
endpoint,
ImmutableMap.of(),
JSON_UTF_8,
@ -144,22 +148,17 @@ class AppEngineConnection implements Connection {
return (Map<String, Object>) JSONValue.parse(response.substring(JSON_SAFETY_PREFIX.length()));
}
@Override
public String getServerUrl() {
return (isLocalhost() ? "http://" : "https://") + getServer().toString();
}
HostAndPort getServer() {
return flags.getServer().withDefaultPort(443); // Default to HTTPS port if unspecified.
}
boolean isLocalhost() {
return flags.getServer().getHost().equals("localhost");
}
private String getUserId() {
return isLocalhost()
? UserIdProvider.getTestUserId()
: UserIdProvider.getProdUserId();
public URL getServer() {
switch (service) {
case DEFAULT:
return RegistryConfig.getDefaultServer();
case TOOLS:
return RegistryConfig.getToolsServer();
case BACKEND:
return RegistryConfig.getBackendServer();
case PUBAPI:
return RegistryConfig.getPubapiServer();
}
throw new IllegalStateException("Unknown service: " + service);
}
}

63
java/google/registry/tools/AppEngineConnectionFlags.java
View file

@ -1,63 +0,0 @@
// Copyright 2017 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.net.HostAndPort;
import dagger.Module;
import dagger.Provides;
import google.registry.config.RegistryConfig;
/**
* Class to contain the configuration flags for AppEngineConnection.
*
* <p>This is broken out into its own class to make it cleaner to extract these from the dagger
* module, where these values are injected.
*/
@Parameters(separators = " =")
class AppEngineConnectionFlags {
@Parameter(names = "--server", description = "HOST[:PORT] to which remote commands are sent.")
private HostAndPort server = RegistryConfig.getServer();
/** Provided for testing. */
@VisibleForTesting
AppEngineConnectionFlags(HostAndPort server) {
this.server = server;
}
AppEngineConnectionFlags() {}
HostAndPort getServer() {
return server;
}
@Module
static class FlagsModule {
AppEngineConnectionFlags flags;
FlagsModule(AppEngineConnectionFlags flags) {
this.flags = flags;
}
@Provides
AppEngineConnectionFlags provideAppEngineConnectionFlags() {
return flags;
}
}
}

4
java/google/registry/tools/BUILD
View file

@ -46,6 +46,7 @@ java_library(
"//java/google/registry/export",
"//java/google/registry/flows",
"//java/google/registry/gcs",
"//java/google/registry/keyring",
"//java/google/registry/keyring/api",
"//java/google/registry/keyring/kms",
"//java/google/registry/loadtest",
@ -69,7 +70,6 @@ java_library(
"@com_google_api_client",
"@com_google_apis_google_api_services_bigquery",
"@com_google_apis_google_api_services_dns",
"@com_google_apis_google_api_services_monitoring",
"@com_google_appengine_api_1_0_sdk",
"@com_google_appengine_remote_api",
"@com_google_appengine_remote_api//:link",
@ -81,8 +81,6 @@ java_library(
"@com_google_guava",
"@com_google_http_client",
"@com_google_http_client_jackson2",
"@com_google_monitoring_client_metrics",
"@com_google_monitoring_client_stackdriver",
"@com_google_oauth_client",
"@com_google_oauth_client_java6",
"@com_google_oauth_client_jetty",

25
java/google/registry/tools/CommandWithConnection.java
View file

@ -14,30 +14,7 @@
package google.registry.tools;
import com.google.common.net.MediaType;
import java.io.IOException;
import java.util.Map;
import javax.annotation.Nullable;
/** A command that can send HTTP requests to a backend module. */
interface CommandWithConnection extends Command {
/** An http connection to AppEngine. */
interface Connection {
void prefetchXsrfToken();
/** Send a POST request. TODO(mmuller): change to sendPostRequest() */
String send(
String endpoint, Map<String, ?> params, MediaType contentType, @Nullable byte[] payload)
throws IOException;
String sendGetRequest(String endpoint, Map<String, ?> params) throws IOException;
Map<String, Object> sendJson(String endpoint, Map<String, ?> object) throws IOException;
String getServerUrl();
}
void setConnection(Connection connection);
void setConnection(AppEngineConnection connection);
}

2
java/google/registry/tools/CreateAnchorTenantCommand.java
View file

@ -25,6 +25,7 @@ import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.net.InternetDomainName;
import com.google.template.soy.data.SoyMapData;
import google.registry.config.RegistryConfig.Config;
import google.registry.tools.soy.CreateAnchorTenantSoyInfo;
import google.registry.util.StringGenerator;
import javax.inject.Inject;
@ -72,6 +73,7 @@ final class CreateAnchorTenantCommand extends MutatingEppToolCommand {
private boolean fee;
@Inject
@Config("base64StringGenerator")
StringGenerator passwordGenerator;
@Override

2
java/google/registry/tools/CreateContactCommand.java
View file

@ -20,6 +20,7 @@ import static com.google.common.base.Strings.isNullOrEmpty;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.template.soy.data.SoyMapData;
import google.registry.config.RegistryConfig.Config;
import google.registry.tools.params.PhoneNumberParameter;
import google.registry.tools.soy.ContactCreateSoyInfo;
import google.registry.util.StringGenerator;
@ -103,6 +104,7 @@ final class CreateContactCommand extends MutatingEppToolCommand {
private String password;
@Inject
@Config("base64StringGenerator")
StringGenerator passwordGenerator;
private static final int PASSWORD_LENGTH = 16;

2
java/google/registry/tools/CreateDomainCommand.java
View file

@ -23,6 +23,7 @@ import static org.joda.time.DateTimeZone.UTC;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.template.soy.data.SoyMapData;
import google.registry.config.RegistryConfig.Config;
import google.registry.model.pricing.PremiumPricingEngine.DomainPrices;
import google.registry.tools.soy.DomainCreateSoyInfo;
import google.registry.util.StringGenerator;
@ -46,6 +47,7 @@ final class CreateDomainCommand extends CreateOrUpdateDomainCommand
private boolean forcePremiums;
@Inject
@Config("base64StringGenerator")
StringGenerator passwordGenerator;
private static final int PASSWORD_LENGTH = 16;

12
java/google/registry/tools/CreateOrUpdatePremiumListCommand.java
View file

@ -57,11 +57,11 @@ abstract class CreateOrUpdatePremiumListCommand extends ConfirmingCommand
required = true)
Path inputFile;
protected Connection connection;
protected AppEngineConnection connection;
protected int inputLineCount;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
@ -101,11 +101,9 @@ abstract class CreateOrUpdatePremiumListCommand extends ConfirmingCommand
}
// Call the server and get the response data
String response = connection.send(
getCommandPath(),
params.build(),
MediaType.FORM_DATA,
requestBody.getBytes(UTF_8));
String response =
connection.sendPostRequest(
getCommandPath(), params.build(), MediaType.FORM_DATA, requestBody.getBytes(UTF_8));
return extractServerResponse(response);
}

4
java/google/registry/tools/CreateRegistrarCommand.java
View file

@ -50,10 +50,10 @@ final class CreateRegistrarCommand extends CreateOrUpdateRegistrarCommand
arity = 1)
boolean createGoogleGroups = true;
private Connection connection;
private AppEngineConnection connection;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}

10
java/google/registry/tools/CreateRegistrarGroupsCommand.java
View file

@ -41,10 +41,10 @@ public class CreateRegistrarGroupsCommand extends ConfirmingCommand
private List<Registrar> registrars = new ArrayList<>();
private Connection connection;
private AppEngineConnection connection;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
@ -66,8 +66,8 @@ public class CreateRegistrarGroupsCommand extends ConfirmingCommand
}
/** Calls the server endpoint to create groups for the specified registrar client id. */
static void executeOnServer(Connection connection, String clientId) throws IOException {
connection.send(
static void executeOnServer(AppEngineConnection connection, String clientId) throws IOException {
connection.sendPostRequest(
CreateGroupsAction.PATH,
ImmutableMap.of(CreateGroupsAction.CLIENT_ID_PARAM, clientId),
MediaType.PLAIN_TEXT_UTF_8,
@ -77,7 +77,7 @@ public class CreateRegistrarGroupsCommand extends ConfirmingCommand
@Override
protected String execute() throws IOException {
for (Registrar registrar : registrars) {
connection.send(
connection.sendPostRequest(
CreateGroupsAction.PATH,
ImmutableMap.of(CreateGroupsAction.CLIENT_ID_PARAM, registrar.getClientId()),
MediaType.PLAIN_TEXT_UTF_8,

21
java/google/registry/tools/CurlCommand.java
View file

@ -22,11 +22,12 @@ import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.net.MediaType;
import google.registry.tools.AppEngineConnection.Service;
import java.util.List;
@Parameters(separators = " =", commandDescription = "Send an HTTP command to the nomulus server.")
class CurlCommand implements CommandWithConnection {
private Connection connection;
private AppEngineConnection connection;
// HTTP Methods that are acceptable for use as values for --method.
public enum Method {
@ -62,8 +63,14 @@ class CurlCommand implements CommandWithConnection {
+ "absent, a GET request is sent.")
private List<String> data;
@Parameter(
names = {"--service"},
description = "Which service to connect to",
required = true)
private Service service;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
@ -77,12 +84,14 @@ class CurlCommand implements CommandWithConnection {
throw new IllegalArgumentException("You may not specify a body for a get method.");
}
// TODO(b/112315418): Make it possible to address any backend.
AppEngineConnection connectionToService = connection.withService(service);
String response =
(method == Method.GET)
? connection.sendGetRequest(path, ImmutableMap.<String, String>of())
: connection.send(
path, ImmutableMap.<String, String>of(), mimeType,
? connectionToService.sendGetRequest(path, ImmutableMap.<String, String>of())
: connectionToService.sendPostRequest(
path,
ImmutableMap.<String, String>of(),
mimeType,
Joiner.on("&").join(data).getBytes(UTF_8));
System.out.println(response);
}

4
java/google/registry/tools/DefaultRequestFactoryModule.java
View file

@ -20,6 +20,7 @@ import com.google.api.client.http.javanet.NetHttpTransport;
import dagger.Binds;
import dagger.Module;
import dagger.Provides;
import google.registry.config.RegistryConfig;
import javax.inject.Named;
import javax.inject.Provider;
@ -43,9 +44,8 @@ class DefaultRequestFactoryModule {
@Provides
@Named("default")
public HttpRequestFactory provideHttpRequestFactory(
AppEngineConnectionFlags connectionFlags,
Provider<Credential> credentialProvider) {
if (connectionFlags.getServer().getHost().equals("localhost")) {
if (RegistryConfig.areServersLocal()) {
return new NetHttpTransport()
.createRequestFactory(
request -> request

107
java/google/registry/tools/DeleteAllocationTokensCommand.java Normal file
View file

@ -0,0 +1,107 @@
// Copyright 2018 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static com.google.common.collect.Iterables.partition;
import static com.google.common.collect.Streams.stream;
import static google.registry.model.ofy.ObjectifyService.ofy;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.domain.token.AllocationToken;
import java.util.List;
/**
* Command to delete unused {@link AllocationToken}s.
*
* <p>Allocation tokens that have been redeemed cannot be deleted. To delete a single allocation
* token, specify the entire token as the prefix.
*/
@Parameters(
separators = " =",
commandDescription = "Deletes the unused AllocationTokens with a given prefix.")
final class DeleteAllocationTokensCommand extends ConfirmingCommand
implements CommandWithRemoteApi {
@Parameter(
names = {"-p", "--prefix"},
description = "Allocation token prefix; if blank, deletes all unused tokens",
required = true)
private String prefix;
@Parameter(
names = {"--with_domains"},
description = "Allow deletion of allocation tokens with specified domains; defaults to false")
boolean withDomains;
@Parameter(
names = {"--dry_run"},
description = "Do not actually delete the tokens; defaults to false")
boolean dryRun;
private static final int BATCH_SIZE = 20;
private static final Joiner JOINER = Joiner.on(", ");
private ImmutableSet<Key<AllocationToken>> tokensToDelete;
@Override
public void init() {
Query<AllocationToken> query =
ofy().load().type(AllocationToken.class).filter("redemptionHistoryEntry", null);
tokensToDelete =
query.keys().list().stream()
.filter(key -> key.getName().startsWith(prefix))
.collect(toImmutableSet());
}
@Override
protected String prompt() {
return String.format(
"Found %d unused tokens starting with '%s' to delete.", tokensToDelete.size(), prefix);
}
@Override
protected String execute() {
long numDeleted =
stream(partition(tokensToDelete, BATCH_SIZE))
.mapToLong(batch -> ofy().transact(() -> deleteBatch(batch)))
.sum();
return String.format("Deleted %d tokens in total.", numDeleted);
}
/** Deletes a (filtered) batch of AllocationTokens and returns how many were deleted. */
private long deleteBatch(List<Key<AllocationToken>> batch) {
// Load the tokens in the same transaction as they are deleted to verify they weren't redeemed
// since the query ran. This also filters out per-domain tokens if they're not to be deleted.
ImmutableSet<AllocationToken> tokensToDelete =
ofy().load().keys(batch).values().stream()
.filter(t -> withDomains || !t.getDomainName().isPresent())
.filter(t -> !t.isRedeemed())
.collect(toImmutableSet());
if (!dryRun) {
ofy().delete().entities(tokensToDelete);
}
System.out.printf(
"%s tokens: %s\n",
dryRun ? "Would delete" : "Deleted",
JOINER.join(batch.stream().map(Key::getName).collect(toImmutableSet())));
return tokensToDelete.size();
}
}

16
java/google/registry/tools/EppToolCommand.java
View file

@ -59,7 +59,7 @@ abstract class EppToolCommand extends ConfirmingCommand
private List<XmlEppParameters> commands = new ArrayList<>();
private Connection connection;
private AppEngineConnection connection;
static class XmlEppParameters {
final String clientId;
@ -95,7 +95,7 @@ abstract class EppToolCommand extends ConfirmingCommand
}
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
@ -145,11 +145,13 @@ abstract class EppToolCommand extends ConfirmingCommand
params.put("xml", URLEncoder.encode(command.xml, UTF_8.toString()));
String requestBody =
Joiner.on('&').withKeyValueSeparator("=").join(filterValues(params, Objects::nonNull));
responses.add(nullToEmpty(connection.send(
"/_dr/epptool",
ImmutableMap.<String, String>of(),
MediaType.FORM_DATA,
requestBody.getBytes(UTF_8))));
responses.add(
nullToEmpty(
connection.sendPostRequest(
"/_dr/epptool",
ImmutableMap.<String, String>of(),
MediaType.FORM_DATA,
requestBody.getBytes(UTF_8))));
}
return responses.build();
}

8
java/google/registry/tools/ExecuteEppCommand.java
View file

@ -20,10 +20,8 @@ import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.io.CharStreams;
import com.google.common.io.Files;
import google.registry.util.NonFinalForTesting;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
@ -40,14 +38,10 @@ final class ExecuteEppCommand extends MutatingEppToolCommand {
required = true)
String clientId;
@NonFinalForTesting
private static InputStream stdin = System.in;
@Override
protected void initMutatingEppToolCommand() throws IOException {
if (mainParameters.isEmpty()) {
addXmlCommand(
clientId, CharStreams.toString(new InputStreamReader(stdin, UTF_8)));
addXmlCommand(clientId, CharStreams.toString(new InputStreamReader(System.in, UTF_8)));
} else {
for (String command : mainParameters) {
addXmlCommand(clientId, Files.asCharSource(new File(command), UTF_8).read());

11
java/google/registry/tools/GenerateAllocationTokensCommand.java
View file

@ -31,6 +31,7 @@ import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableSet;
import com.google.common.io.Files;
import com.googlecode.objectify.Key;
import google.registry.config.RegistryConfig.Config;
import google.registry.model.domain.token.AllocationToken;
import google.registry.util.NonFinalForTesting;
import google.registry.util.Retrier;
@ -40,16 +41,15 @@ import java.io.IOException;
import java.util.Collection;
import java.util.Deque;
import javax.inject.Inject;
import javax.inject.Named;
/** Command to generate and persist {@link AllocationToken}s. */
@NonFinalForTesting
@Parameters(
separators = " =",
commandDescription =
"Generates and persists the given number of AllocationTokens, printing each token to stdout."
)
public class GenerateAllocationTokensCommand implements CommandWithRemoteApi {
@NonFinalForTesting
class GenerateAllocationTokensCommand implements CommandWithRemoteApi {
@Parameter(
names = {"-p", "--prefix"},
@ -80,7 +80,10 @@ public class GenerateAllocationTokensCommand implements CommandWithRemoteApi {
description = "Do not actually persist the tokens; defaults to false")
boolean dryRun;
@Inject @Named("base58StringGenerator") StringGenerator stringGenerator;
@Inject
@Config("base58StringGenerator")
StringGenerator stringGenerator;
@Inject Retrier retrier;
private static final int BATCH_SIZE = 20;

28
java/google/registry/tools/GenerateLordnCommand.java
View file

@ -22,7 +22,7 @@ import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.collect.ImmutableList;
import google.registry.model.domain.DomainResource;
import google.registry.tmch.LordnTask;
import google.registry.tmch.LordnTaskUtils;
import google.registry.tools.params.PathParameter;
import java.io.IOException;
import java.nio.file.Files;
@ -61,26 +61,28 @@ final class GenerateLordnCommand implements CommandWithRemoteApi {
for (DomainResource domain : ofy().load().type(DomainResource.class).filter("tld", tld)) {
String status = " ";
if (domain.getLaunchNotice() == null && domain.getSmdId() != null) {
sunriseCsv.add(LordnTask.getCsvLineForSunriseDomain(domain, domain.getCreationTime()));
sunriseCsv.add(LordnTaskUtils.getCsvLineForSunriseDomain(domain, domain.getCreationTime()));
status = "S";
} else if (domain.getLaunchNotice() != null || domain.getSmdId() != null) {
claimsCsv.add(LordnTask.getCsvLineForClaimsDomain(domain, domain.getCreationTime()));
claimsCsv.add(LordnTaskUtils.getCsvLineForClaimsDomain(domain, domain.getCreationTime()));
status = "C";
}
System.out.printf("%s[%s] ", domain.getFullyQualifiedDomainName(), status);
}
ImmutableList<String> claimsRows = claimsCsv.build();
ImmutableList<String> claimsAll = new ImmutableList.Builder<String>()
.add(String.format("1,%s,%d", now, claimsRows.size()))
.add(LordnTask.COLUMNS_CLAIMS)
.addAll(claimsRows)
.build();
ImmutableList<String> claimsAll =
new ImmutableList.Builder<String>()
.add(String.format("1,%s,%d", now, claimsRows.size()))
.add(LordnTaskUtils.COLUMNS_CLAIMS)
.addAll(claimsRows)
.build();
ImmutableList<String> sunriseRows = sunriseCsv.build();
ImmutableList<String> sunriseAll = new ImmutableList.Builder<String>()
.add(String.format("1,%s,%d", now.plusMillis(1), sunriseRows.size()))
.add(LordnTask.COLUMNS_SUNRISE)
.addAll(sunriseRows)
.build();
ImmutableList<String> sunriseAll =
new ImmutableList.Builder<String>()
.add(String.format("1,%s,%d", now.plusMillis(1), sunriseRows.size()))
.add(LordnTaskUtils.COLUMNS_SUNRISE)
.addAll(sunriseRows)
.build();
Files.write(claimsOutputPath, claimsAll, UTF_8);
Files.write(sunriseOutputPath, sunriseAll, UTF_8);
}

9
java/google/registry/tools/GenerateZoneFilesCommand.java
View file

@ -45,10 +45,10 @@ final class GenerateZoneFilesCommand implements CommandWithConnection, CommandWi
validateWith = DateParameter.class)
private DateTime exportDate = DateTime.now(UTC).minus(standardMinutes(2)).withTimeAtStartOfDay();
private Connection connection;
private AppEngineConnection connection;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
@ -59,10 +59,7 @@ final class GenerateZoneFilesCommand implements CommandWithConnection, CommandWi
"tlds", mainParameters,
"exportTime", exportDate.toString());
Map<String, Object> response = connection.sendJson(GenerateZoneFilesAction.PATH, params);
System.out.printf(
"Job started at %s%s\n",
connection.getServerUrl(),
response.get("jobPath"));
System.out.printf("Job started at %s %s\n", connection.getServer(), response.get("jobPath"));
System.out.println("Output files:");
@SuppressWarnings("unchecked")
List<String> filenames = (List<String>) response.get("filenames");

10
java/google/registry/tools/ListDomainsCommand.java
View file

@ -15,11 +15,14 @@
package google.registry.tools;
import static com.google.common.base.Preconditions.checkArgument;
import static google.registry.model.registry.Registries.getTldsOfType;
import static google.registry.util.CollectionUtils.isNullOrEmpty;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableMap;
import google.registry.model.registry.Registry.TldType;
import google.registry.tools.server.ListDomainsAction;
import java.util.List;
@ -29,8 +32,7 @@ final class ListDomainsCommand extends ListObjectsCommand {
@Parameter(
names = {"-t", "--tld", "--tlds"},
description = "Comma-delimited list of top-level domain(s) to list second-level domains of.",
required = true)
description = "Comma-delimited list of TLDs to list domains on; defaults to all REAL TLDs.")
private List<String> tlds;
@Parameter(
@ -47,6 +49,10 @@ final class ListDomainsCommand extends ListObjectsCommand {
/** Returns a map of parameters to be sent to the server (in addition to the usual ones). */
@Override
ImmutableMap<String, Object> getParameterMap() {
// Default to all REAL TLDs if not specified.
if (isNullOrEmpty(tlds)) {
tlds = getTldsOfType(TldType.REAL).asList();
}
String tldsParam = Joiner.on(',').join(tlds);
checkArgument(tldsParam.length() < 1024, "Total length of TLDs is too long for URL parameter");
return ImmutableMap.of("tlds", tldsParam, "limit", maxDomains);

12
java/google/registry/tools/ListObjectsCommand.java
View file

@ -54,10 +54,10 @@ abstract class ListObjectsCommand implements CommandWithConnection, CommandWithR
description = "Whether to print full field names in header row (as opposed to aliases)")
private boolean fullFieldNames = false;
private Connection connection;
private AppEngineConnection connection;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
@ -83,11 +83,9 @@ abstract class ListObjectsCommand implements CommandWithConnection, CommandWithR
}
params.putAll(getParameterMap());
// Call the server and get the response data.
String response = connection.send(
getCommandPath(),
params.build(),
MediaType.PLAIN_TEXT_UTF_8,
new byte[0]);
String response =
connection.sendPostRequest(
getCommandPath(), params.build(), MediaType.PLAIN_TEXT_UTF_8, new byte[0]);
// Parse the returned JSON and make sure it's a map.
Object obj = JSONValue.parse(response.substring(JSON_SAFETY_PREFIX.length()));
if (!(obj instanceof Map<?, ?>)) {

11
java/google/registry/tools/LoadTestCommand.java
View file

@ -77,10 +77,10 @@ class LoadTestCommand extends ConfirmingCommand
description = "Time to run the load test in seconds.")
int runSeconds = DEFAULT_RUN_SECONDS;
private Connection connection;
private AppEngineConnection connection;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
@ -127,10 +127,7 @@ class LoadTestCommand extends ConfirmingCommand
.put("runSeconds", runSeconds)
.build();
return connection.send(
LoadTestAction.PATH,
params,
MediaType.PLAIN_TEXT_UTF_8,
new byte[0]);
return connection.sendPostRequest(
LoadTestAction.PATH, params, MediaType.PLAIN_TEXT_UTF_8, new byte[0]);
}
}

53
java/google/registry/tools/MetricToolModule.java
View file

@ -1,53 +0,0 @@
// Copyright 2018 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.services.monitoring.v3.Monitoring;
import com.google.api.services.monitoring.v3.model.MonitoredResource;
import com.google.monitoring.metrics.MetricWriter;
import com.google.monitoring.metrics.stackdriver.StackdriverWriter;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule.DefaultCredential;
import google.registry.config.RegistryConfig.Config;
/** Dagger module for metrics on the client tool. */
@Module
public final class MetricToolModule {
@Provides
static Monitoring provideMonitoring(
@DefaultCredential GoogleCredential credential, @Config("projectId") String projectId) {
return new Monitoring.Builder(
credential.getTransport(), credential.getJsonFactory(), credential)
.setApplicationName(projectId)
.build();
}
@Provides
static MetricWriter provideMetricWriter(
Monitoring monitoringClient,
@Config("projectId") String projectId,
@Config("stackdriverMaxQps") int maxQps,
@Config("stackdriverMaxPointsPerRequest") int maxPointsPerRequest) {
return new StackdriverWriter(
monitoringClient,
projectId,
new MonitoredResource().setType("global"),
maxQps,
maxPointsPerRequest);
}
}

12
java/google/registry/tools/RegistrarContactCommand.java
View file

@ -57,19 +57,19 @@ final class RegistrarContactCommand extends MutatingCommand {
@Parameter(
description = "Client identifier of the registrar account.",
required = true)
private List<String> mainParameters;
List<String> mainParameters;
@Parameter(
names = "--mode",
description = "Type of operation you want to perform (LIST, CREATE, UPDATE, or DELETE).",
required = true)
private Mode mode;
Mode mode;
@Nullable
@Parameter(
names = "--name",
description = "Contact name.")
private String name;
String name;
@Nullable
@Parameter(
@ -82,7 +82,7 @@ final class RegistrarContactCommand extends MutatingCommand {
@Parameter(
names = "--email",
description = "Contact email address.")
private String email;
String email;
@Nullable
@Parameter(
@ -105,7 +105,7 @@ final class RegistrarContactCommand extends MutatingCommand {
names = "--allow_console_access",
description = "Enable or disable access to the registrar console for this contact.",
arity = 1)
private Boolean allowConsoleAccess;
Boolean allowConsoleAccess;
@Nullable
@Parameter(
@ -138,7 +138,7 @@ final class RegistrarContactCommand extends MutatingCommand {
validateWith = PathParameter.OutputFile.class)
private Path output = Paths.get("/dev/stdout");
private enum Mode { LIST, CREATE, UPDATE, DELETE }
enum Mode { LIST, CREATE, UPDATE, DELETE }
private static final ImmutableSet<Mode> MODES_REQUIRING_CONTACT_SYNC =
ImmutableSet.of(Mode.CREATE, Mode.UPDATE, Mode.DELETE);

90
java/google/registry/tools/RegistryCli.java
View file

@ -26,19 +26,11 @@ import com.beust.jcommander.Parameter;
import com.beust.jcommander.ParameterException;
import com.beust.jcommander.Parameters;
import com.beust.jcommander.ParametersDelegate;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Iterables;
import com.google.monitoring.metrics.IncrementableMetric;
import com.google.monitoring.metrics.LabelDescriptor;
import com.google.monitoring.metrics.Metric;
import com.google.monitoring.metrics.MetricPoint;
import com.google.monitoring.metrics.MetricRegistryImpl;
import com.google.monitoring.metrics.MetricWriter;
import google.registry.config.RegistryConfig;
import google.registry.model.ofy.ObjectifyService;
import google.registry.tools.params.ParameterFactory;
import java.io.IOException;
import java.security.Security;
import java.util.Map;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
@ -62,14 +54,6 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
description = "Returns all command names.")
private boolean showAllCommands;
@VisibleForTesting
boolean uploadMetrics = true;
// Do not make this final - compile-time constant inlining may interfere with JCommander.
@ParametersDelegate
private AppEngineConnectionFlags appEngineConnectionFlags =
new AppEngineConnectionFlags();
// Do not make this final - compile-time constant inlining may interfere with JCommander.
@ParametersDelegate
@ -85,24 +69,6 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
// "shell".
private boolean isFirstUse = true;
private static final ImmutableSet<LabelDescriptor> LABEL_DESCRIPTORS_FOR_COMMANDS =
ImmutableSet.of(
LabelDescriptor.create("program", "The program used - e.g. nomulus or gtech_tool"),
LabelDescriptor.create("environment", "The environment used - e.g. sandbox"),
LabelDescriptor.create("command", "The command used"),
LabelDescriptor.create("success", "Whether the command succeeded"),
LabelDescriptor.create("shell", "Whether the command was called from the nomulus shell"));
private static final IncrementableMetric commandsCalledCount =
MetricRegistryImpl.getDefault()
.newIncrementableMetric(
"/tools/commands_called",
"Count of tool commands called",
"count",
LABEL_DESCRIPTORS_FOR_COMMANDS);
private MetricWriter metricWriter = null;
Map<String, ? extends Class<? extends Command>> commands;
String programName;
@ -114,7 +80,6 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
Security.addProvider(new BouncyCastleProvider());
component = DaggerRegistryToolComponent.builder()
.flagsModule(new AppEngineConnectionFlags.FlagsModule(appEngineConnectionFlags))
.build();
}
@ -124,13 +89,9 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
// http://www.angelikalanger.com/GenericsFAQ/FAQSections/TypeArguments.html#FAQ104
@Override
public void run(String[] args) throws Exception {
boolean inShell = !isFirstUse;
isFirstUse = false;
// Create the JCommander instance.
// If we're in the shell, we don't want to update the RegistryCli's parameters (so we give a
// dummy object to update)
JCommander jcommander = new JCommander(inShell ? new Object() : this);
JCommander jcommander = new JCommander(this);
jcommander.addConverterFactory(new ParameterFactory());
jcommander.setProgramName(programName);
@ -149,8 +110,8 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
// Create the "help" and "shell" commands (these are special in that they don't have a default
// constructor).
jcommander.addCommand("help", new HelpCommand(jcommander));
if (!inShell) {
// If we aren't inside a shell, then we want to add the shell command.
if (isFirstUse) {
isFirstUse = false;
ShellCommand shellCommand = new ShellCommand(this);
// We have to build the completions based on the jcommander *before* we add the "shell"
// command - to avoid completion for the "shell" command itself.
@ -192,31 +153,17 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
jcommander.getCommands().get(jcommander.getParsedCommand()).getObjects());
loggingParams.configureLogging(); // Must be called after parameters are parsed.
boolean success = false;
try {
runCommand(command);
success = true;
} catch (AuthModule.LoginRequiredException ex) {
System.err.println("===================================================================");
System.err.println("You must login using 'nomulus login' prior to running this command.");
System.err.println("===================================================================");
} finally {
commandsCalledCount.increment(
programName,
environment.toString(),
command.getClass().getSimpleName(),
String.valueOf(success),
String.valueOf(inShell));
exportMetrics();
}
}
@Override
public void close() {
exportMetrics();
if (metricWriter != null) {
metricWriter = null;
}
if (installer != null) {
installer.uninstall();
installer = null;
@ -233,14 +180,6 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
private void runCommand(Command command) throws Exception {
injectReflectively(RegistryToolComponent.class, component, command);
if (metricWriter == null && uploadMetrics) {
try {
metricWriter = component.metricWriter();
} catch (Exception e) {
System.err.format("Failed to get metricWriter. Got error:\n%s\n\n", e);
uploadMetrics = false;
}
}
if (command instanceof CommandWithConnection) {
((CommandWithConnection) command).setConnection(getConnection());
@ -253,7 +192,7 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
RemoteApiOptions options = new RemoteApiOptions();
options.server(
getConnection().getServer().getHost(), getConnection().getServer().getPort());
if (getConnection().isLocalhost()) {
if (RegistryConfig.areServersLocal()) {
// Use dev credentials for localhost.
options.useDevelopmentServerCredential();
} else {
@ -272,25 +211,6 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
command.run();
}
private void exportMetrics() {
if (metricWriter == null) {
return;
}
try {
for (Metric<?> metric : MetricRegistryImpl.getDefault().getRegisteredMetrics()) {
for (MetricPoint<?> point : metric.getTimestampedValues()) {
metricWriter.write(point);
}
}
metricWriter.flush();
} catch (IOException e) {
System.err.format("Failed to export metrics. Got error:\n%s\n\n", e);
System.err.println("Maybe you need to login? Try calling:");
System.err.println(" gcloud auth application-default login");
}
}
@VisibleForTesting
void setEnvironment(RegistryToolEnvironment environment) {
this.environment = environment;
}

1
java/google/registry/tools/RegistryTool.java
View file

@ -48,6 +48,7 @@ public final class RegistryTool {
.put("create_sandbox_tld", CreateSandboxTldCommand.class)
.put("create_tld", CreateTldCommand.class)
.put("curl", CurlCommand.class)
.put("delete_allocation_tokens", DeleteAllocationTokensCommand.class)
.put("delete_domain", DeleteDomainCommand.class)
.put("delete_host", DeleteHostCommand.class)
.put("delete_premium_list", DeletePremiumListCommand.class)

20
java/google/registry/tools/RegistryToolComponent.java
View file

@ -14,7 +14,6 @@
package google.registry.tools;
import com.google.monitoring.metrics.MetricWriter;
import dagger.Component;
import google.registry.bigquery.BigqueryModule;
import google.registry.config.CredentialModule;
@ -22,17 +21,15 @@ import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.dns.writer.VoidDnsWriterModule;
import google.registry.dns.writer.clouddns.CloudDnsWriterModule;
import google.registry.dns.writer.dnsupdate.DnsUpdateWriterModule;
import google.registry.keyring.KeyringModule;
import google.registry.keyring.api.DummyKeyringModule;
import google.registry.keyring.api.KeyModule;
import google.registry.keyring.kms.KmsModule;
import google.registry.rde.RdeModule;
import google.registry.request.Modules.AppIdentityCredentialModule;
import google.registry.request.Modules.DatastoreServiceModule;
import google.registry.request.Modules.GoogleCredentialModule;
import google.registry.request.Modules.Jackson2Module;
import google.registry.request.Modules.NetHttpTransportModule;
import google.registry.request.Modules.URLFetchServiceModule;
import google.registry.request.Modules.UrlFetchTransportModule;
import google.registry.request.Modules.UseAppIdentityCredentialForGoogleApisModule;
import google.registry.request.Modules.UserServiceModule;
import google.registry.util.AppEngineServiceUtilsImpl.AppEngineServiceUtilsModule;
import google.registry.util.SystemClock.SystemClockModule;
@ -49,37 +46,30 @@ import javax.inject.Singleton;
@Singleton
@Component(
modules = {
AppEngineConnectionFlags.FlagsModule.class,
AppEngineServiceUtilsModule.class,
// TODO(b/36866706): Find a way to replace this with a command-line friendly version
AppIdentityCredentialModule.class,
AuthModule.class,
BigqueryModule.class,
ConfigModule.class,
CredentialModule.class,
DatastoreServiceModule.class,
google.registry.keyring.api.DummyKeyringModule.class,
DummyKeyringModule.class,
CloudDnsWriterModule.class,
DefaultRequestFactoryModule.class,
DefaultRequestFactoryModule.RequestFactoryModule.class,
DnsUpdateWriterModule.class,
GoogleCredentialModule.class,
Jackson2Module.class,
KeyModule.class,
KeyringModule.class,
KmsModule.class,
NetHttpTransportModule.class,
RdeModule.class,
RegistryToolModule.class,
SystemClockModule.class,
SystemSleeperModule.class,
URLFetchServiceModule.class,
UrlFetchTransportModule.class,
// TODO(b/36866706): Find a way to replace this with a command-line friendly version
UseAppIdentityCredentialForGoogleApisModule.class,
UserServiceModule.class,
VoidDnsWriterModule.class,
WhoisModule.class,
MetricToolModule.class,
})
interface RegistryToolComponent {
void inject(CheckDomainClaimsCommand command);
@ -118,6 +108,4 @@ interface RegistryToolComponent {
void inject(WhoisQueryCommand command);
AppEngineConnection appEngineConnection();
MetricWriter metricWriter();
}

67
java/google/registry/tools/RegistryToolModule.java
View file

@ -1,67 +0,0 @@
// Copyright 2017 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools;
import dagger.Binds;
import dagger.Module;
import dagger.Provides;
import google.registry.util.RandomStringGenerator;
import google.registry.util.StringGenerator;
import google.registry.util.StringGenerator.Alphabets;
import java.security.NoSuchAlgorithmException;
import java.security.ProviderException;
import java.security.SecureRandom;
import javax.inject.Named;
/** Dagger module for Registry Tool. */
@Module
abstract class RegistryToolModule {
@Provides
static RegistryToolEnvironment provideRegistryToolEnvironment() {
return RegistryToolEnvironment.get();
}
@Binds
abstract StringGenerator provideStringGenerator(RandomStringGenerator stringGenerator);
@Provides
static SecureRandom provideSecureRandom() {
try {
return SecureRandom.getInstance("NativePRNG");
} catch (NoSuchAlgorithmException e) {
throw new ProviderException(e);
}
}
@Provides
@Named("alphabetBase64")
static String provideAlphabetBase64() {
return Alphabets.BASE_64;
}
@Provides
@Named("alphabetBase58")
static String provideAlphabetBase58() {
return Alphabets.BASE_58;
}
@Provides
@Named("base58StringGenerator")
static StringGenerator provideBase58StringGenerator(
@Named("alphabetBase58") String alphabet, SecureRandom random) {
return new RandomStringGenerator(alphabet, random);
}
}

75
java/google/registry/tools/SetupOteCommand.java
View file

@ -15,15 +15,21 @@
package google.registry.tools;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkNotNull;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.tools.CommandUtilities.promptForYes;
import static google.registry.util.X509Utils.loadCertificate;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Throwables;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSortedMap;
import com.google.re2j.Pattern;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryEnvironment;
import google.registry.model.common.GaeUserIdConverter;
import google.registry.model.registrar.Registrar;
import google.registry.model.registry.Registry.TldState;
import google.registry.tools.params.PathParameter;
@ -53,6 +59,10 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
private static final Duration SHORT_REDEMPTION_GRACE_PERIOD = Duration.standardMinutes(10);
private static final Duration SHORT_PENDING_DELETE_LENGTH = Duration.standardMinutes(5);
// Whether to prompt the user on command failures. Set to false for testing of these failures.
@VisibleForTesting
static boolean interactive = true;
private static final ImmutableSortedMap<DateTime, Money> EAP_FEE_SCHEDULE =
ImmutableSortedMap.of(
new DateTime(0),
@ -87,6 +97,14 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
)
private List<String> ipWhitelist = new ArrayList<>();
@Parameter(
names = {"--email"},
description =
"the registrar's account to use for console access. "
+ "Must be on the registry's G-Suite domain.",
required = true)
private String email;
@Parameter(
names = {"-c", "--certfile"},
description = "full path to cert file in PEM format (best if on local storage)",
@ -122,7 +140,9 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
)
private boolean eapOnly = false;
@Inject StringGenerator passwordGenerator;
@Inject
@Config("base64StringGenerator")
StringGenerator passwordGenerator;
/**
* Long registrar names are truncated and then have an incrementing digit appended at the end so
@ -130,6 +150,21 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
*/
private int roidSuffixCounter = 0;
/** Runs a command, clearing the cache before and prompting the user on failures. */
private void runCommand(Command command) {
ofy().clearSessionCache();
try {
command.run();
} catch (Exception e) {
System.err.format("Command failed with error %s\n", e);
if (interactive && promptForYes("Continue to next command?")) {
return;
}
Throwables.throwIfUnchecked(e);
throw new RuntimeException(e);
}
}
/** Constructs and runs a CreateTldCommand. */
private void createTld(
String tldName,
@ -137,8 +172,7 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
Duration addGracePeriod,
Duration redemptionGracePeriod,
Duration pendingDeleteLength,
boolean isEarlyAccess)
throws Exception {
boolean isEarlyAccess) {
CreateTldCommand command = new CreateTldCommand();
command.addGracePeriod = addGracePeriod;
command.dnsWriters = dnsWriters;
@ -158,11 +192,11 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
if (isEarlyAccess) {
command.eapFeeSchedule = EAP_FEE_SCHEDULE;
}
command.run();
runCommand(command);
}
/** Constructs and runs a CreateRegistrarCommand */
private void createRegistrar(String registrarName, String password, String tld) throws Exception {
private void createRegistrar(String registrarName, String password, String tld) {
CreateRegistrarCommand command = new CreateRegistrarCommand();
command.mainParameters = ImmutableList.of(registrarName);
command.createGoogleGroups = false; // Don't create Google Groups for OT&E registrars.
@ -183,7 +217,19 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
command.phone = Optional.of("+1.2125550100");
command.icannReferralEmail = "nightmare@registrar.test";
command.force = force;
command.run();
runCommand(command);
}
/** Constructs and runs a RegistrarContactCommand */
private void createRegistrarContact(String registrarName) {
RegistrarContactCommand command = new RegistrarContactCommand();
command.mainParameters = ImmutableList.of(registrarName);
command.mode = RegistrarContactCommand.Mode.CREATE;
command.name = email;
command.email = email;
command.allowConsoleAccess = true;
command.force = force;
runCommand(command);
}
/** Run any pre-execute command checks */
@ -193,6 +239,14 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
REGISTRAR_PATTERN.matcher(registrar).matches(),
"Registrar name is invalid (see usage text for requirements).");
// Make sure the email is "correct" - as in it's a valid email we can convert to gaeId
// There's no need to look at the result - it'll be converted again inside
// RegistrarContactCommand.
checkNotNull(
GaeUserIdConverter.convertEmailAddressToGaeUserId(email),
"Email address %s is not associated with any GAE ID",
email);
boolean warned = false;
if (RegistryEnvironment.get() != RegistryEnvironment.SANDBOX
&& RegistryEnvironment.get() != RegistryEnvironment.UNITTEST) {
@ -227,7 +281,9 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
return "Creating TLD:\n"
+ " " + registrar + "-eap\n"
+ "Creating registrar:\n"
+ " " + registrar + "-5 (access to TLD " + registrar + "-eap)";
+ " " + registrar + "-5 (access to TLD " + registrar + "-eap)\n"
+ "Giving contact access to this registrar:\n"
+ " " + email;
} else {
return "Creating TLDs:\n"
+ " " + registrar + "-sunrise\n"
@ -239,7 +295,9 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
+ " " + registrar + "-2 (access to TLD " + registrar + "-landrush)\n"
+ " " + registrar + "-3 (access to TLD " + registrar + "-ga)\n"
+ " " + registrar + "-4 (access to TLD " + registrar + "-ga)\n"
+ " " + registrar + "-5 (access to TLD " + registrar + "-eap)";
+ " " + registrar + "-5 (access to TLD " + registrar + "-eap)\n"
+ "Giving contact access to these registrars:\n"
+ " " + email;
}
}
@ -297,6 +355,7 @@ final class SetupOteCommand extends ConfirmingCommand implements CommandWithRemo
for (List<String> r : registrars) {
createRegistrar(r.get(0), r.get(1), r.get(2));
createRegistrarContact(r.get(0));
}
StringBuilder output = new StringBuilder();

145
java/google/registry/tools/ShellCommand.java
View file

@ -145,6 +145,80 @@ public class ShellCommand implements Command {
return this;
}
private static class OutputEncapsulator {
private PrintStream orgStdout;
private PrintStream orgStderr;
private EncapsulatingOutputStream encapsulatedOutputStream = null;
private EncapsulatingOutputStream encapsulatedErrorStream = null;
private Exception error;
private OutputEncapsulator() {
orgStdout = System.out;
orgStderr = System.err;
encapsulatedOutputStream = new EncapsulatingOutputStream(System.out, "out: ");
encapsulatedErrorStream = new EncapsulatingOutputStream(System.out, "err: ");
System.setOut(new PrintStream(encapsulatedOutputStream));
System.setErr(new PrintStream(encapsulatedErrorStream));
}
void setError(Exception e) {
error = e;
}
private void restoreOriginalStreams() {
try {
encapsulatedOutputStream.dumpLastLine();
encapsulatedErrorStream.dumpLastLine();
System.setOut(orgStdout);
System.setErr(orgStderr);
if (error != null) {
emitFailure(error);
} else {
emitSuccess();
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
/**
* Emit a success command separator.
*
* <p>Dumps the last line of output prior to doing this.
*/
private void emitSuccess() {
System.out.println(SUCCESS);
System.out.flush();
}
/**
* Emit a failure message obtained from the throwable.
*
* <p>Dumps the last line of output prior to doing this.
*/
private void emitFailure(Throwable e) {
System.out.println(
FAILURE
+ e.getClass().getName()
+ " "
+ e.getMessage().replace("\\", "\\\\").replace("\n", "\\n"));
}
/** Run "func" with output encapsulation. */
static void run(CommandRunner runner, String[] args) {
OutputEncapsulator encapsulator = new OutputEncapsulator();
try {
runner.run(args);
} catch (Exception e) {
encapsulator.setError(e);
} finally {
encapsulator.restoreOriginalStreams();
}
}
}
/** Run the shell until the user presses "Ctrl-D". */
@Override
public void run() {
@ -154,23 +228,6 @@ public class ShellCommand implements Command {
String line;
DateTime lastTime = clock.nowUtc();
while ((line = getLine()) != null) {
PrintStream orgStdout = null;
PrintStream orgStderr = null;
EncapsulatingOutputStream encapsulatedOutputStream = null;
EncapsulatingOutputStream encapsulatedErrorStream = null;
// Wrap standard output and error if requested. We have to do so here in run because the flags
// haven't been processed in the constructor.
if (encapsulateOutput) {
orgStdout = System.out;
orgStderr = System.err;
encapsulatedOutputStream = new EncapsulatingOutputStream(System.out, "out: ");
encapsulatedErrorStream = new EncapsulatingOutputStream(System.out, "err: ");
System.setOut(new PrintStream(encapsulatedOutputStream));
System.setErr(new PrintStream(encapsulatedErrorStream));
}
// Make sure we're not idle for too long. Only relevant when we're "extra careful"
if (!dontExitOnIdle
&& beExtraCareful
@ -184,28 +241,17 @@ public class ShellCommand implements Command {
if (lineArgs.length == 0) {
continue;
}
Exception lastError = null;
try {
System.out.println("Barf!!!");
runner.run(lineArgs);
} catch (Exception e) {
lastError = e;
System.err.println("Got an exception:\n" + e);
}
try {
if (encapsulatedOutputStream != null) {
encapsulatedOutputStream.dumpLastLine();
encapsulatedErrorStream.dumpLastLine();
System.setOut(orgStdout);
System.setErr(orgStderr);
if (lastError == null) {
emitSuccess();
} else {
emitFailure(lastError);
}
// Wrap standard output and error if requested. We have to do so here in run because the flags
// haven't been processed in the constructor.
if (encapsulateOutput) {
OutputEncapsulator.run(runner, lineArgs);
} else {
try {
runner.run(lineArgs);
} catch (Exception e) {
System.err.println("Got an exception:\n" + e);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
if (!encapsulateOutput) {
@ -244,29 +290,6 @@ public class ShellCommand implements Command {
return resultBuilder.build().toArray(new String[0]);
}
/**
* Emit a success command separator.
*
* <p>Dumps the last line of output prior to doing this.
*/
private void emitSuccess() {
System.out.println(SUCCESS);
System.out.flush();
}
/**
* Emit a failure message obtained from the throwable.
*
* <p>Dumps the last line of output prior to doing this.
*/
private void emitFailure(Throwable e) {
System.out.println(
FAILURE
+ e.getClass().getName()
+ " "
+ e.getMessage().replace("\\", "\\\\").replace("\n", "\\n"));
}
@VisibleForTesting
static class JCommanderCompletor implements Completor {

4
java/google/registry/tools/VerifyOteCommand.java
View file

@ -57,10 +57,10 @@ final class VerifyOteCommand implements CommandWithConnection, CommandWithRemote
description = "Only show a summary of information")
private boolean summarize;
private Connection connection;
private AppEngineConnection connection;
@Override
public void setConnection(Connection connection) {
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}

3
java/google/registry/tools/logging.properties
View file

@ -2,7 +2,4 @@ handlers = java.util.logging.ConsoleHandler
.level = INFO
com.google.wrappers.base.GoogleInit.level = WARNING
com.google.monitoring.metrics.MetricRegistryImpl.level = WARNING
com.google.monitoring.metrics.MetricReporter.level = WARNING
com.google.monitoring.metrics.MetricExporter.level = WARNING
com.google.monitoring.metrics.stackdriver.StackdriverWriter.level = WARNING

11
java/google/registry/ui/js/registrar/console.js
View file

@ -14,11 +14,13 @@
goog.provide('registry.registrar.Console');
goog.require('goog.Uri');
goog.require('goog.dispose');
goog.require('goog.dom');
goog.require('goog.dom.classlist');
goog.require('goog.net.XhrIo');
goog.require('registry.Console');
goog.require('registry.Resource');
goog.require('registry.registrar.Contact');
goog.require('registry.registrar.ContactSettings');
goog.require('registry.registrar.ContactUs');
@ -76,7 +78,7 @@ registry.registrar.Console = function(params) {
/**
* @type {!Object.<string, function(new:registry.Component,
* !registry.registrar.Console,
* string)>}
* !registry.Resource)>}
*/
this.pageMap = {};
this.pageMap['security-settings'] = registry.registrar.SecuritySettings;
@ -136,7 +138,10 @@ registry.registrar.Console.prototype.handleHashChange = function() {
componentCtor = this.pageMap[''];
}
var oldComponent = this.component_;
this.component_ = new componentCtor(this, this.params.xsrfToken);
const resource = new registry.Resource(
new goog.Uri('/registrar-settings'), this.params.clientId,
this.params.xsrfToken);
this.component_ = new componentCtor(this, resource);
this.registerDisposable(this.component_);
this.component_.basePath = type;
this.component_.bindToDom(id);
@ -155,7 +160,7 @@ registry.registrar.Console.prototype.changeNavStyle = function() {
slashNdx = slashNdx == -1 ? hashToken.length : slashNdx;
var regNavlist = goog.dom.getRequiredElement('reg-navlist');
var path = hashToken.substring(0, slashNdx);
var active = regNavlist.querySelector('a[href="/registrar#' + path + '"]');
var active = regNavlist.querySelector('a[href="#' + path + '"]');
if (goog.isNull(active)) {
registry.util.log('Unknown path or path form in changeNavStyle.');
return;

12
java/google/registry/ui/js/registrar/contact_settings.js
View file

@ -14,7 +14,6 @@
goog.provide('registry.registrar.ContactSettings');
goog.require('goog.Uri');
goog.require('goog.array');
goog.require('goog.dom');
goog.require('goog.dom.TagName');
@ -38,18 +37,15 @@ goog.forwardDeclare('registry.registrar.Console');
* updating only that field of the Registrar object and not
* implementing the create action from edit_item.
* @param {!registry.registrar.Console} console
* @param {string} xsrfToken Security token to pass back to the server.
* @param {!registry.Resource} resource the RESTful resource for the registrar.
* @constructor
* @extends {registry.ResourceComponent}
* @final
*/
registry.registrar.ContactSettings = function(console, xsrfToken) {
registry.registrar.ContactSettings = function(console, resource) {
registry.registrar.ContactSettings.base(
this, 'constructor',
console,
new registry.Resource(new goog.Uri('/registrar-settings'), xsrfToken),
registry.soy.registrar.contacts.contact,
null);
this, 'constructor', console, resource,
registry.soy.registrar.contacts.contact, null);
};
goog.inherits(registry.registrar.ContactSettings, registry.ResourceComponent);

Some files were not shown because too many files have changed in this diff Show more