This makes the deletion mapreduce more resilient in the face of data integrity
violations (which exist on sandbox but hopefully not in production). Even when
the domain application index doesn't exist, we still want to delete the domain
application itself, as its continuing presence will cause problems after the
code for domain applications is deleted.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228521794
Two commands are being added:
- ImportDatastoreCommand starts an async import operation.
User may choose to wait until import completes or quit
immediately.
- GetOperationStatusCommand checks the status of an operation.
It may be used to check the status of an operation started by
ImportDatastoreCommand.
Both commands communicate with Datastore admin api directly, without
going through the Registry server.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228400059
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout. Instead copy directly from the decoder
to standard output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
It was saying it was deleting tokens it wasn't, because it was outputting the
raw input list of tokens rather than the list that filtered out redeemed or
domain-specific tokens.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227769266
This also deletes associated entities including indexes and history entries.
This needs to run as a prerequisite to [] which deletes all domain
application code entirely. The entities themselves need to be deleted first so
that loading DomainBases in the future doesn't accidentally get applications
which the code no longer knows how to handle.
This deletion is safe to perform because the only remaining applications in our
system are historical and we no longer refer to them. Backups will be retained
in BigQuery.
This mapreduce will be deleted at the same time that the DomainApplication code
is.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227738528
Currently, you have to set "--ip_whitelist=null", which is unintuitive.
This adds the option to just give an empty string: "--ip_whitelist="
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227550896
This does not change existing functionality but will allow us to use this common code in the yet-to-be-created web console action as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227159346
This allows us to run nomulus tool programmatically on environments that do not
allow the 3-legged OAuth authentication flow.
The provided JSON file corresponds to a service account, which must have
GAE admin permission and whose client ID must be whitelisted in the config
file.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226008337
This is in preparation for having a web-console endpoint to create OTE.
In addition - we streamline the code:
- we remove support for different premium lists
- we remove support for different DNS writers - we never want a "real" DnsWriter for OTE
- we remove support of --eap_only, because we don't need it anymore
- We use a single password for all the Registrars
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225841694
This is safer and addresses a common source of confusion in the codebase because it's always explicit that the resource returned may not be present, whether because it's soft-deleted when projected to the given time or because it never existed in the first place.
In production code, the presence of the returned value is always checked. In test code, its presence is assumed using .get() where that is expected and convenient, as it not being present will throw an NPE that will cause the test to fail anyway.
Note that the roughly equivalent reloadResourceByForeignKey(), which is widely used in test code, is not having this same treatment applied to it. That is out of the scope of this CL, and has much smaller returns anyway because it's only used in tests (where the unexpected absence of a given resource would just cause the test to fail).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225424002
This will allow us to perform the OT&E history verification
in the model/ package as well so that it can be used both
by both the UI and the command line tool.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225007167
This is used to reduce the expiration time of domain(s) by some number of years
(if enough length remains in the registration term to do so). This does not back
out the previously saved BillingEvent entities as they may have already been
sent out and invoiced, so any related refunds must be handled out of band.
In addition to reducing the registration expiration time on the domain itself,
this command writes out a new history entry, one-time poll message informing the
registrar of this change, auto-renew billing event and poll message, and
updates/ends the old auto-renew billing event and poll message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224999285
We are moving away from using Application Default Credentials generated by "gcloud auth application-default login" in our code base and consolidate on using self-managed credentials provided from AuthModule.
One of the remaining dependencies on the ADCs is from beam pipeline deployment commands, which by default use the ADCs to talk to GCS and upload the jar files and templates. In this CL, we explicitly provide the locally created credential to the Options used in deployments.
Also moved all credential qualifiers to CredentialModule, and removed @AppEngineAdminApiCredential, which is no longer used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224199812
The scenarios in which the credential is used are:
1) Calls to Nomulus GAE HTTP endpoints.
2) Calls to Google APIs within the tool.
3) Calls to GAE APIs within the tool.
From now on the tool should not depend on ADCs created from gcloud any more (expect for beam pipeline deployments which need some more investigation as the dependency on ADC is not apparent). Using the nomulus tool requires running "nomulus login" first, but only once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224165735
RemoteApiOption has a package-private method that takes a Stream representing the content of a JSON and use a GoogleCredential created from it as its credential. This CL uses reflection to change the access modifier of that method in order to supply a credential stream that is self-managed. This is obviously not ideal and prone to breakage in case the getGoogleCredentialStream method is changed. Unfortunately upstream is not willing to make it public citing the reason that GoogleCredential.fromStream() (which getGoogleCredentialStream uses) is a @Beta annotated function (see https://groups.google.com[]forum/#!searchin/domain-registry-eng/remoteapioptions%7Csort:date/domain-registry-eng/Flsah6skszQ/CySZv2XEBwAJ). However this function is introduced 5 years ago as a public function (b857184bfa). I think at this point it is safe to assume that it is part of the widely used APIs and will not change without sufficient notice.
Note here that RemoteApiOptions creates its own copy of GoogleCredential to be used to call App Engine APIs locally, whereas communications to Nomulus endpoints use the Credential provided in AuthModule. Even though both credentials are created from the same client id, client secret and refresh token (the three elements needed to construct a GoogleCredential this way, see https://github.com/googleapis/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/GoogleCredential.java#L842), their refreshes cycles are independent of each other. I verified that refreshing one of the credential does not invalidate the access token of the other credential, as long as it is not expired yet.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224156131
content-type needs to be parsed (no automatic parsing from String)
data was splitting on commas, meaning --data="key=value1,value2" was sent to the server as "key=value1&value2"
NOTE - you'd expect there to already be a "do nothing splitter", right? But there isn't :/
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223346887
As previously written, it loaded up all history entries into memory and then
processed them. This was OOMing for some registrars on sandbox who had performed
a large number of testing actions, most of them long OT&E was passed.
This commit changes the verify OT&E action to stream the history entries in
batches, ordered by when they were made, and then terminates once all tests have
passed. This prevents OOMing because only a single batch of history entries need
reside in memory at once.
This does necessitate the creation of a new composite Datastore index on
HistoryEntry, so we'll need to run the ResaveAllHistoryEntriesAction in sandbox
after this change is deployed before the new verify OT&E code will work.
Note that the "history viewer" is long dead, but that the pre-existing index
on HistoryEntries is still used for many other purposes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223163337
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
Many registry tools tests modify system properties but do not
restore them to original state. These tests must be isolated
from each other and cannot share the same test execution process.
This has a huge impact on test performance under Gradle, which
seems to have higher process startup overhead. Current Gradle
test config has to set 'forEvery' to 1, i.e., every test class
must be run in a freshly started process.
This change significantly reduces the number of tests that need
isolation, making it easier to optimize test config for the
remaining tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221350284
for a given service and version at runtime.
Note that this CL only supports the adjustment for a given service and version. I will add another functionality to let this command be able to detect all non-live versions automatically and apply the adjustment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221092001
This is in preparation for having other "command changing things" like
redirecting to file and maybe variable substitutions in the arguments.
"On the way" added a
RUNNING "some_command" "--some_flag" "some_value"
to the output encapsulator so that if we run multiple commands, we know what
command was called where.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219837452
Move the shell output encapsulation so that we don't double-wrap on a
premature exit.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219136896
AppEngineConnection can now connect to all services and not just the tools.
The default is still the tools.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218734983
The main goal of this CL is to make the clock constructor injectable (so that tests do not need to use InjectRule to override the clock). The complication is that the clock is used by two static caches (ROOT_CACHE directly and CRL_CACHE indirectly). The clock is not actually used to construct the lock, but rather to verify that the root certificate is within its validity period.
For ROOT_CACHE we move the verification to its call sites. This adds a bit overhead because the validity check happens every time the cache is called, not just when the cache is built or refreshed. However this check is rather cheap. Also the resources are included in the jar and the cache is valid for 1 year. Given that we deploy every week, there's not much point making it an expiring cache rather than a static map.
For CRL_CACHE we change the key to a tuple of TmchCaMode and X509Certificate. The certificate is no longer provided from the ROOT_CACHE directly and must be verified before it is provided as a cache key. We left the CRL verification inside the cache loader because it (signature verification) is more expensive compared to simple expiration check, and we do not want to do this every time the cache is called.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218385684
When creating the various registrar objects in Sandbox for OTE, we also give access to all the registrars' data to a given google account (identified by the email)
This email has to belong to the registry's G-Suite account, just like in the registrar_contact command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217728407
This is what you want most of the time when you're running the command to
see recent domains, as this will list all the real billable domains and
exclude the prober ones.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217603275
*** Reason for rollback ***
Automated tools sometimes don't have default credentials, and can't set them up. We should redo this CL once we figure out the credential thing.
*** Original change description ***
Add metrics for the command used in the registry CLI tool
Puts the metric in <project>/tools/commands_called
It counts the use of the tool, with the following labels:
- environment
- tool (nomulus/gtech)
- command called (class name)
- success true/false
- from the shell true/false
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=214048616
This change completes the switch to @DefaultCredential for
all use cases in GAE.
Impacted modules:
- IcannReporting
- CreateCdnsTld command
- LoadSnapshot command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=213511730
It doesn't do anything that ALLOWED_IN_SUNRISE doesn't do, and there's no point
in having two separate types when we can simply keep track of the semantic
difference between the two by using different lists (as we have for .soy).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212997752
Puts the metric in <project>/tools/commands_called
It counts the use of the tool, with the following labels:
- environment
- tool (nomulus/gtech)
- command called (class name)
- success true/false
- from the shell true/false
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212879670
The vast majority of the time this is the registrar client ID you want, so
there's no reason to require specifying it everything each time. These are
read-only commands anyway, so the potential negative effects are minimal.
See the existing lock/unlock_domain commands for existing occurrences of this
behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211857712
This change required several things:
- Separating out the interfaces that merely do HTTP calls to the backend from those
that require the remote API (only load the remote API for the latter). Only the
tools service provides the remote api endpoint.
- Removing the XSRF token as an authentication mechanism (with OAUTH, we no longer
need this, and trying to provide it requires initialization of the datastore
code which requires the remote API)
I can't think of a compelling unit test for this beyond what already exists.
Tested:
Verified that:
- nomulus tool commands (e.g. "list_tlds") work against the tools service as they
currently do
- The "curl" command hits endpoints on "tools" by default.
- We can use --server to specify endpoints on the default service.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211510454
Do not allow the user to create TLDs on sandbox that aren't of the form
"*.test.". If real TLDs are created, they will block users from registering
names under that TLD for the nameserver set that we're using for sandbox.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209983482
We never used it and don't have any plans to use it going forward. All
conceivable parts of its functionality that we might use going forward have
already been subsumed into allocation tokens, which are a simpler way of
handling the same use case that are also standards-compliant.
Also gets rid of the hideous ANCHOR_ prefix on anchor tenant EPP authcodes
that was only ever necessary because of overloading the authcode for
anchor tenant creation. Going forward it'll be based on allocation tokens,
so there's no risk of conflicts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209418194
16 is consistent with how we've generated codes for anchor tenants in the past.
Also gets rid of a space in the output so that it's a fully valid CSV.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208106631
The design doc is at []
The next step will be to tie this into the domain create flow, and if the domain
name is on a reserved list, allow it to be created if the token is specified that
has the given domain name on it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207884521
Create a command to send arbitrary, authenticated HTTP requests to the backend
and remove the existing commands that are basically just wrappers around this.
Tested:
In addition to the unit tests, verified both get and post requests against
alpha.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207756509
This is left over from the transition from Guava to Java 8 Optionals.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207154260
Also adjusts the nomulus list_cursors command to output the value of this field.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206646117
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).
This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.
This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
