This temporarily disables the ability to download service account credentials. We'd like this to go in to next week's release so that we do not expose this functionality to users who otherwise do not have the ability to download the service account credential, while we research on a more secure solution.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223397754
content-type needs to be parsed (no automatic parsing from String)
data was splitting on commas, meaning --data="key=value1,value2" was sent to the server as "key=value1&value2"
NOTE - you'd expect there to already be a "do nothing splitter", right? But there isn't :/
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223346887
As previously written, it loaded up all history entries into memory and then
processed them. This was OOMing for some registrars on sandbox who had performed
a large number of testing actions, most of them long OT&E was passed.
This commit changes the verify OT&E action to stream the history entries in
batches, ordered by when they were made, and then terminates once all tests have
passed. This prevents OOMing because only a single batch of history entries need
reside in memory at once.
This does necessitate the creation of a new composite Datastore index on
HistoryEntry, so we'll need to run the ResaveAllHistoryEntriesAction in sandbox
after this change is deployed before the new verify OT&E code will work.
Note that the "history viewer" is long dead, but that the pre-existing index
on HistoryEntries is still used for many other purposes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223163337
These targets are removed in [] as they are not compile-time
dependencies. But they are needed at runtime by GetRoutingMapCommand to load
the specific classes to generate routing map for.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222916432
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
Currently, OTE creates an invalid address "state", which doesn't pass validation when trying to update it on the console.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221372006
Many registry tools tests modify system properties but do not
restore them to original state. These tests must be isolated
from each other and cannot share the same test execution process.
This has a huge impact on test performance under Gradle, which
seems to have higher process startup overhead. Current Gradle
test config has to set 'forEvery' to 1, i.e., every test class
must be run in a freshly started process.
This change significantly reduces the number of tests that need
isolation, making it easier to optimize test config for the
remaining tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221350284
for a given service and version at runtime.
Note that this CL only supports the adjustment for a given service and version. I will add another functionality to let this command be able to detect all non-live versions automatically and apply the adjustment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221092001
This is in preparation for having other "command changing things" like
redirecting to file and maybe variable substitutions in the arguments.
"On the way" added a
RUNNING "some_command" "--some_flag" "some_value"
to the output encapsulator so that if we run multiple commands, we know what
command was called where.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219837452
Move the shell output encapsulation so that we don't double-wrap on a
premature exit.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219136896
AppEngineConnection can now connect to all services and not just the tools.
The default is still the tools.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218734983
Make every dependency request explicit on what encoding is used. Also get rid of InjectRule in XjcToDomainResourceConverterTest.
Random number generator providers are separated to secure and insecure ones. The insecure ones must be explicitly requested (usually for use cases where security is not of concern, for better speed).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217921422
Made it clear that it is a util class and moved some of the functions only used in NordnUploadAction (to NordnUploadAction). Also used Retrier to handle retries when leasing tasks.
These changes allow us to no longer use InjectRule in related unit tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217761117
When creating the various registrar objects in Sandbox for OTE, we also give access to all the registrars' data to a given google account (identified by the email)
This email has to belong to the registry's G-Suite account, just like in the registrar_contact command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217728407
This is what you want most of the time when you're running the command to
see recent domains, as this will list all the real billable domains and
exclude the prober ones.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217603275
This uses a Dagger-provided map of Keyring implementations, with two currently available,
"KMS" and "Dummy". The active keyring is configured in the YAML file, so we no longer
require MOE directives to choose which one to use for internal/external builds.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=216898058
All credentials provided by these modules have been
replaced by those in the config/CredentialsModule,
with a new set of Qualifiers. With Dagger 2, a successful
build means that the removal is safe.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=215258792
*** Reason for rollback ***
Automated tools sometimes don't have default credentials, and can't set them up. We should redo this CL once we figure out the credential thing.
*** Original change description ***
Add metrics for the command used in the registry CLI tool
Puts the metric in <project>/tools/commands_called
It counts the use of the tool, with the following labels:
- environment
- tool (nomulus/gtech)
- command called (class name)
- success true/false
- from the shell true/false
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=214048616
This change completes the switch to @DefaultCredential for
all use cases in GAE.
Impacted modules:
- IcannReporting
- CreateCdnsTld command
- LoadSnapshot command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=213511730
This is only useful when we used the [] proxy because the GFE requires SNI during handshake in order to request the client certificate. The GCP proxy does not need this (it always requests the client certificate). We do not need to check for its existence.
Also removed the checking of internal headers for ssl cert hash used only by the [] proxy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=213059027
Puts the metric in <project>/tools/commands_called
It counts the use of the tool, with the following labels:
- environment
- tool (nomulus/gtech)
- command called (class name)
- success true/false
- from the shell true/false
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212879670
The vast majority of the time this is the registrar client ID you want, so
there's no reason to require specifying it everything each time. These are
read-only commands anyway, so the potential negative effects are minimal.
See the existing lock/unlock_domain commands for existing occurrences of this
behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211857712
As the first step in credential consolidation, we replace
injection of application default credential in for KMS and
Drive.
Tests:
- for Drive, tested with exportDomainLists and exportReservedTerms.
- For KMS, used CLI commands (get_keyring_secret and update_kms_keyring) to change and
restore secret for one key.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211819859
ServerSideCommand now just aggregates CommandWithConnection and
CommandWithRemoteApi, so it's arguably clearer for commands to just implement
both of these.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211670031
This change required several things:
- Separating out the interfaces that merely do HTTP calls to the backend from those
that require the remote API (only load the remote API for the latter). Only the
tools service provides the remote api endpoint.
- Removing the XSRF token as an authentication mechanism (with OAUTH, we no longer
need this, and trying to provide it requires initialization of the datastore
code which requires the remote API)
I can't think of a compelling unit test for this beyond what already exists.
Tested:
Verified that:
- nomulus tool commands (e.g. "list_tlds") work against the tools service as they
currently do
- The "curl" command hits endpoints on "tools" by default.
- We can use --server to specify endpoints on the default service.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211510454
Do not allow the user to create TLDs on sandbox that aren't of the form
"*.test.". If real TLDs are created, they will block users from registering
names under that TLD for the nameserver set that we're using for sandbox.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209983482
We never used it and don't have any plans to use it going forward. All
conceivable parts of its functionality that we might use going forward have
already been subsumed into allocation tokens, which are a simpler way of
handling the same use case that are also standards-compliant.
Also gets rid of the hideous ANCHOR_ prefix on anchor tenant EPP authcodes
that was only ever necessary because of overloading the authcode for
anchor tenant creation. Going forward it'll be based on allocation tokens,
so there's no risk of conflicts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209418194
Removing this stanza from the config will cause sandbox to write to production
cloud dns, which is what we want.
Likewise, exclude sandbox in addition to production in the create_cdns_tld
command from the environments that point to staging.
Cloud DNS has 3 environments that we would consider using:
- staging which is reset every week, so we can't use it for sandbox
- testing, which is not accessible from external App Engine
- production
Because of the difficulties with the first two, we've decided to use production.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208834786
16 is consistent with how we've generated codes for anchor tenants in the past.
Also gets rid of a space in the output so that it's a fully valid CSV.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208106631
This adds actual subdomain verification via the SafeBrowsing API to the Spec11
pipeline, as well as on-the-fly KMS decryption via the GenerateSpec11Action to
securely store our API key in source code.
Testing the interaction becomes difficult due to serialization requirements, and will be significantly expanded in the next cl. For now, it verifies basic end-to-end pipeline behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208092942
The design doc is at []
The next step will be to tie this into the domain create flow, and if the domain
name is on a reserved list, allow it to be created if the token is specified that
has the given domain name on it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207884521
Create a command to send arbitrary, authenticated HTTP requests to the backend
and remove the existing commands that are basically just wrappers around this.
Tested:
In addition to the unit tests, verified both get and post requests against
alpha.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207756509
Also adjusts the nomulus list_cursors command to output the value of this field.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206646117
non-compliant packages that depend on java_x_proto_library targets. This will enable blaze
to enforce strict_deps by default while missing dependencies are added to these packages.
Changes made using newly released blaze flag:
USE_CANARY_BLAZE=nightly blaze build -k --experimental_java_proto_library_enforce_strict_deps
then extracting the packages from the resulting add_dep commands, and for each package running:
buildozer 'add features -jpl_strict_deps' <package>:__pkg__
More information: []
Tested:
TAP sample presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206349847
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).
This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.
This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
This adds the scaffolding for a basic Spec11 pipeline- it gathers all domains from all time for a given project and counts how many there are. I've factored out a few common utilities for beam pipelines to avoid excessive duplication.
Future CLs will:
- Actually process domains via the SafeBrowsing API
- Generate a real spec11 report
- Template queries based on the input YearMonth
- Abstract more commonalities across beam pipelines to reduce boilerplate when adding new pipelines.
TESTED: FOSS test passed, and ran successfully on alpha
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205997741
I'm finally fed up enough with all the nameserver changes we've had to make on our
self-allocated domains to improve the command. Now you can simply run:
$ nomulus ... update_domain ... -n ns[1-4].foo.bar
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205282317
This prepares for the spec11 beam pipeline to live parallel to the invoicing
beam pipeline, for better organization.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204980582
Second step of RDE encoding refactoring.
Creates a single OutputStream encode RyDE files.
This replaces the 5 OutputStreams that were needed before.
Also removes all the factories that were injected. It's an encoding, there's no point in injecting it.
Finally, removed the buffer-size configuration and replaced with a static final
const value in each individual OutputStream.
This doesn't yet include a decoder (InputStream). And there's still a lot of overlap between the Ryde and the Ghostryde code. Both of those are left for the next CLs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204898369
Lists used as accumulators were being updated individually for each domain
without starting over from a fresh list each time, so the number of changes
would grow for each additional domain and potentially be wrong if the previous
domains were set up differently.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204526006
