This makes the deletion mapreduce more resilient in the face of data integrity
violations (which exist on sandbox but hopefully not in production). Even when
the domain application index doesn't exist, we still want to delete the domain
application itself, as its continuing presence will cause problems after the
code for domain applications is deleted.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228521794
Two commands are being added:
- ImportDatastoreCommand starts an async import operation.
User may choose to wait until import completes or quit
immediately.
- GetOperationStatusCommand checks the status of an operation.
It may be used to check the status of an operation started by
ImportDatastoreCommand.
Both commands communicate with Datastore admin api directly, without
going through the Registry server.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228400059
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout. Instead copy directly from the decoder
to standard output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout. Instead copy directly from the decoder
to standard output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
It was saying it was deleting tokens it wasn't, because it was outputting the
raw input list of tokens rather than the list that filtered out redeemed or
domain-specific tokens.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227769266
This also deletes associated entities including indexes and history entries.
This needs to run as a prerequisite to [] which deletes all domain
application code entirely. The entities themselves need to be deleted first so
that loading DomainBases in the future doesn't accidentally get applications
which the code no longer knows how to handle.
This deletion is safe to perform because the only remaining applications in our
system are historical and we no longer refer to them. Backups will be retained
in BigQuery.
This mapreduce will be deleted at the same time that the DomainApplication code
is.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227738528
Currently, you have to set "--ip_whitelist=null", which is unintuitive.
This adds the option to just give an empty string: "--ip_whitelist="
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227550896
This does not change existing functionality but will allow us to use this common code in the yet-to-be-created web console action as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227159346
Most dependencies on the Old SDK were switched in [] This is just catching up on some OAuth dependencies that remained and some remaining uses of Old build rules.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226337284
This allows us to run nomulus tool programmatically on environments that do not
allow the 3-legged OAuth authentication flow.
The provided JSON file corresponds to a service account, which must have
GAE admin permission and whose client ID must be whitelisted in the config
file.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226008337
This is in preparation for having a web-console endpoint to create OTE.
In addition - we streamline the code:
- we remove support for different premium lists
- we remove support for different DNS writers - we never want a "real" DnsWriter for OTE
- we remove support of --eap_only, because we don't need it anymore
- We use a single password for all the Registrars
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225841694
This is safer and addresses a common source of confusion in the codebase because it's always explicit that the resource returned may not be present, whether because it's soft-deleted when projected to the given time or because it never existed in the first place.
In production code, the presence of the returned value is always checked. In test code, its presence is assumed using .get() where that is expected and convenient, as it not being present will throw an NPE that will cause the test to fail anyway.
Note that the roughly equivalent reloadResourceByForeignKey(), which is widely used in test code, is not having this same treatment applied to it. That is out of the scope of this CL, and has much smaller returns anyway because it's only used in tests (where the unexpected absence of a given resource would just cause the test to fail).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225424002
If the user runs "nomulus -e [ENV] login --remote", an URL will be provided, the user then can visit the URL on any machine (not necessary where the command is run) and copy&paste back the authorization code to complete authorization.
This makes it easy to login on machines where local browsers are not easily accessible.
Also upgraded nebula lint version to 10.3.5.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225198700
Note that it's possible to set a config option to disable this functionality
on a per-environment basis (we're disabling it for sandbox), but in general
SSL certificate hashes should be required for increased security.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225053496
This will allow us to perform the OT&E history verification
in the model/ package as well so that it can be used both
by both the UI and the command line tool.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225007167
This is used to reduce the expiration time of domain(s) by some number of years
(if enough length remains in the registration term to do so). This does not back
out the previously saved BillingEvent entities as they may have already been
sent out and invoiced, so any related refunds must be handled out of band.
In addition to reducing the registration expiration time on the domain itself,
this command writes out a new history entry, one-time poll message informing the
registrar of this change, auto-renew billing event and poll message, and
updates/ends the old auto-renew billing event and poll message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224999285
We are moving away from using Application Default Credentials generated by "gcloud auth application-default login" in our code base and consolidate on using self-managed credentials provided from AuthModule.
One of the remaining dependencies on the ADCs is from beam pipeline deployment commands, which by default use the ADCs to talk to GCS and upload the jar files and templates. In this CL, we explicitly provide the locally created credential to the Options used in deployments.
Also moved all credential qualifiers to CredentialModule, and removed @AppEngineAdminApiCredential, which is no longer used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224199812
The scenarios in which the credential is used are:
1) Calls to Nomulus GAE HTTP endpoints.
2) Calls to Google APIs within the tool.
3) Calls to GAE APIs within the tool.
From now on the tool should not depend on ADCs created from gcloud any more (expect for beam pipeline deployments which need some more investigation as the dependency on ADC is not apparent). Using the nomulus tool requires running "nomulus login" first, but only once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224165735
RemoteApiOption has a package-private method that takes a Stream representing the content of a JSON and use a GoogleCredential created from it as its credential. This CL uses reflection to change the access modifier of that method in order to supply a credential stream that is self-managed. This is obviously not ideal and prone to breakage in case the getGoogleCredentialStream method is changed. Unfortunately upstream is not willing to make it public citing the reason that GoogleCredential.fromStream() (which getGoogleCredentialStream uses) is a @Beta annotated function (see https://groups.google.com[]forum/#!searchin/domain-registry-eng/remoteapioptions%7Csort:date/domain-registry-eng/Flsah6skszQ/CySZv2XEBwAJ). However this function is introduced 5 years ago as a public function (b857184bfa). I think at this point it is safe to assume that it is part of the widely used APIs and will not change without sufficient notice.
Note here that RemoteApiOptions creates its own copy of GoogleCredential to be used to call App Engine APIs locally, whereas communications to Nomulus endpoints use the Credential provided in AuthModule. Even though both credentials are created from the same client id, client secret and refresh token (the three elements needed to construct a GoogleCredential this way, see https://github.com/googleapis/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/GoogleCredential.java#L842), their refreshes cycles are independent of each other. I verified that refreshing one of the credential does not invalidate the access token of the other credential, as long as it is not expired yet.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224156131
This temporarily disables the ability to download service account credentials. We'd like this to go in to next week's release so that we do not expose this functionality to users who otherwise do not have the ability to download the service account credential, while we research on a more secure solution.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223397754
content-type needs to be parsed (no automatic parsing from String)
data was splitting on commas, meaning --data="key=value1,value2" was sent to the server as "key=value1&value2"
NOTE - you'd expect there to already be a "do nothing splitter", right? But there isn't :/
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223346887
As previously written, it loaded up all history entries into memory and then
processed them. This was OOMing for some registrars on sandbox who had performed
a large number of testing actions, most of them long OT&E was passed.
This commit changes the verify OT&E action to stream the history entries in
batches, ordered by when they were made, and then terminates once all tests have
passed. This prevents OOMing because only a single batch of history entries need
reside in memory at once.
This does necessitate the creation of a new composite Datastore index on
HistoryEntry, so we'll need to run the ResaveAllHistoryEntriesAction in sandbox
after this change is deployed before the new verify OT&E code will work.
Note that the "history viewer" is long dead, but that the pre-existing index
on HistoryEntries is still used for many other purposes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223163337
These targets are removed in [] as they are not compile-time
dependencies. But they are needed at runtime by GetRoutingMapCommand to load
the specific classes to generate routing map for.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222916432
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
Currently, OTE creates an invalid address "state", which doesn't pass validation when trying to update it on the console.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221372006
Many registry tools tests modify system properties but do not
restore them to original state. These tests must be isolated
from each other and cannot share the same test execution process.
This has a huge impact on test performance under Gradle, which
seems to have higher process startup overhead. Current Gradle
test config has to set 'forEvery' to 1, i.e., every test class
must be run in a freshly started process.
This change significantly reduces the number of tests that need
isolation, making it easier to optimize test config for the
remaining tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221350284
for a given service and version at runtime.
Note that this CL only supports the adjustment for a given service and version. I will add another functionality to let this command be able to detect all non-live versions automatically and apply the adjustment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221092001
This is in preparation for having other "command changing things" like
redirecting to file and maybe variable substitutions in the arguments.
"On the way" added a
RUNNING "some_command" "--some_flag" "some_value"
to the output encapsulator so that if we run multiple commands, we know what
command was called where.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219837452
Move the shell output encapsulation so that we don't double-wrap on a
premature exit.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219136896
AppEngineConnection can now connect to all services and not just the tools.
The default is still the tools.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218734983
Make every dependency request explicit on what encoding is used. Also get rid of InjectRule in XjcToDomainResourceConverterTest.
Random number generator providers are separated to secure and insecure ones. The insecure ones must be explicitly requested (usually for use cases where security is not of concern, for better speed).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217921422
Made it clear that it is a util class and moved some of the functions only used in NordnUploadAction (to NordnUploadAction). Also used Retrier to handle retries when leasing tasks.
These changes allow us to no longer use InjectRule in related unit tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217761117
When creating the various registrar objects in Sandbox for OTE, we also give access to all the registrars' data to a given google account (identified by the email)
This email has to belong to the registry's G-Suite account, just like in the registrar_contact command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217728407
This is what you want most of the time when you're running the command to
see recent domains, as this will list all the real billable domains and
exclude the prober ones.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217603275
This uses a Dagger-provided map of Keyring implementations, with two currently available,
"KMS" and "Dummy". The active keyring is configured in the YAML file, so we no longer
require MOE directives to choose which one to use for internal/external builds.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=216898058
All credentials provided by these modules have been
replaced by those in the config/CredentialsModule,
with a new set of Qualifiers. With Dagger 2, a successful
build means that the removal is safe.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=215258792
*** Reason for rollback ***
Automated tools sometimes don't have default credentials, and can't set them up. We should redo this CL once we figure out the credential thing.
*** Original change description ***
Add metrics for the command used in the registry CLI tool
Puts the metric in <project>/tools/commands_called
It counts the use of the tool, with the following labels:
- environment
- tool (nomulus/gtech)
- command called (class name)
- success true/false
- from the shell true/false
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=214048616
