If the user runs "nomulus -e [ENV] login --remote", an URL will be provided, the user then can visit the URL on any machine (not necessary where the command is run) and copy&paste back the authorization code to complete authorization.
This makes it easy to login on machines where local browsers are not easily accessible.
Also upgraded nebula lint version to 10.3.5.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225198700
Tested:
TAP --sample for global presubmit queue
[] dr-foss-test passes
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225189790
Next up (and a much larger commit) will be giving loadByForeignKey() the same
treatment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225182377
Note that it's possible to set a config option to disable this functionality
on a per-environment basis (we're disabling it for sandbox), but in general
SSL certificate hashes should be required for increased security.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225053496
This will allow us to perform the OT&E history verification
in the model/ package as well so that it can be used both
by both the UI and the command line tool.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225007167
This is used to reduce the expiration time of domain(s) by some number of years
(if enough length remains in the registration term to do so). This does not back
out the previously saved BillingEvent entities as they may have already been
sent out and invoiced, so any related refunds must be handled out of band.
In addition to reducing the registration expiration time on the domain itself,
this command writes out a new history entry, one-time poll message informing the
registrar of this change, auto-renew billing event and poll message, and
updates/ends the old auto-renew billing event and poll message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224999285
Add server end points to backup Datastore using managed-export mechanism.
A cron job is defined in Alpha to run daily exports using this implementation.
Existing backup is left running. The new backups are saved to a new set of
locations:
- GCS bucket: gs://PROJECT-ID-datastore-backups
- Big Query data set: datastore_backups
- Big Query latest back up view name: latest_datastore_backup
Also, the names of Bigquery tables now use the export timestamp
assigned by Datastore. E.g., 2018_12_05T23_56_18_50532_ContactResource,
After the new import mechanism is implemented and the back-restore flow is
tested, we will stop the existing backup runs and deploy the new
implementation to all environments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224932957
Eventually the Publish action will control daily/monthly sending and provide
the correct threats to email. The goal of this PR is to entirely separate
the "sending email" functionality from the "parsing threat matches"
functionality.
The PublishAction will figure out if the monthly emails should be sent out,
then will ask the Spec11ThreatMatchesParser for the monthly threats (if
appropriate) and the new threat matches for today. It will then pass those
matches and the appropriate email subject+body to the email utils class,
whose only job is to format and send the emails.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224869643
We have a retrier wanting to call a function, retrying on
IllegalStateExceptions (thrown by checkState()).
However, one of the checkStates is called inside a Concurrent.transform, so
when the checkState fails, the resulting IllegalStateException is wrapped in an
UncheckedExecutionException and isn't caught by the retrier.
We unwrap the IllegalStateException to make sure it's caught.
Q: Why not just catch UncheckedExecution exception?
A: Because it might wrap a different exception which we don't want to retry on.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224862573
The lastEppUpdateTime should be updated asynchronously in the situations below:
- Implicit transfer success after 5 day pending transfer period
- Implicit end of any grace period
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224831791
It'd be nice if we can separate out the tool to its own package and reduce the transitive dependencies that it pulls in. However since the entire core project is a dependency of the tool, it doesn't make any difference as we'd be pulling in core and all its transitive dependencies as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224821931
When a connection to the proxy using the PROXY protocol (https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) comes from an IP address that the external load balancer does not recognize, make the source IP 0.0.0.0. This way an appropriate WHOIS quota can be configured for this kind of connections.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224583547
This is for consistency, mostly the LocalDate fields added in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224525451
Add a sendSpec11Email parameter that allows us to only send the email on
one run per month. Next, we will compute the diffs between the daily runs
and send daily emails with those diffs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224404653
Removed duplicate build files for each service, and also added two tasks that runs deployment/staging for all services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224359198
We are moving away from using Application Default Credentials generated by "gcloud auth application-default login" in our code base and consolidate on using self-managed credentials provided from AuthModule.
One of the remaining dependencies on the ADCs is from beam pipeline deployment commands, which by default use the ADCs to talk to GCS and upload the jar files and templates. In this CL, we explicitly provide the locally created credential to the Options used in deployments.
Also moved all credential qualifiers to CredentialModule, and removed @AppEngineAdminApiCredential, which is no longer used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224199812
The scenarios in which the credential is used are:
1) Calls to Nomulus GAE HTTP endpoints.
2) Calls to Google APIs within the tool.
3) Calls to GAE APIs within the tool.
From now on the tool should not depend on ADCs created from gcloud any more (expect for beam pipeline deployments which need some more investigation as the dependency on ADC is not apparent). Using the nomulus tool requires running "nomulus login" first, but only once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224165735
RemoteApiOption has a package-private method that takes a Stream representing the content of a JSON and use a GoogleCredential created from it as its credential. This CL uses reflection to change the access modifier of that method in order to supply a credential stream that is self-managed. This is obviously not ideal and prone to breakage in case the getGoogleCredentialStream method is changed. Unfortunately upstream is not willing to make it public citing the reason that GoogleCredential.fromStream() (which getGoogleCredentialStream uses) is a @Beta annotated function (see https://groups.google.com[]forum/#!searchin/domain-registry-eng/remoteapioptions%7Csort:date/domain-registry-eng/Flsah6skszQ/CySZv2XEBwAJ). However this function is introduced 5 years ago as a public function (b857184bfa). I think at this point it is safe to assume that it is part of the widely used APIs and will not change without sufficient notice.
Note here that RemoteApiOptions creates its own copy of GoogleCredential to be used to call App Engine APIs locally, whereas communications to Nomulus endpoints use the Credential provided in AuthModule. Even though both credentials are created from the same client id, client secret and refresh token (the three elements needed to construct a GoogleCredential this way, see https://github.com/googleapis/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/GoogleCredential.java#L842), their refreshes cycles are independent of each other. I verified that refreshing one of the credential does not invalidate the access token of the other credential, as long as it is not expired yet.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224156131
Also moved pushing tags to the private repo to the last step as we do not want
to tag the configs unless the build is successful.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223982622
Now that we're shipping gradlew, use it from travis. This is the prescribed
approach in the travis docs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223976622
Also updated distroless base image to include the latest vulnerability fixes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223975505
The version of gradle used in compilation is now controlled by the gradle
wrapper and is centrally managed. This means that everyone will be using the
same version of gradle for build, including IC systems like Travis and Cloud
Build.
The Gradle version for this Gradle wrapper is v5.0.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223856122
Currently the /registrar-settings backend endpoint will fail to update any
OWNER fields that a non-OWNER tries to change.
However, the front-end (soy, js) still allow non-OWNERs to try and change
these fields (there's the "edit" or "add" button, and it only fails when you try to "save")
This CL changes the front-end to remove the ability for non-OWNERs to even try
and change these fields. However, it will still let them *view* these fields as
it has interesting and important information.
-------------------------------
In addition - it changes the webdriver tests to include the "edit buttons". Those were never tested before, and now we will test to see if they are indeed displayed or not.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223845883
Remove the application and version tags and replace "module" with "service".
The old tags have been deprecated for ages, and they cause warnings when we
do gradle deployment/staging.
TESTED:
Deployed to crash, everything still seemed to work to the extent that crash
can be expected to, service variations still happened, all services got
updated. Minimal risk to the other environments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223799777
This also raises their access level so they can be called by other classes that
extend EppTestCase (which I'm writing in a follow-up CL).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223512618
This temporarily disables the ability to download service account credentials. We'd like this to go in to next week's release so that we do not expose this functionality to users who otherwise do not have the ability to download the service account credential, while we research on a more secure solution.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223397754
Add a first pass add Nomulus GAE deployment to our gradle build scripts. This
attempt could stand some improvement. In particular:
- It currently only deploys to crash, per discussion we will make the
environment a parameter.
- This uses a different import mechanism from the other plugins (via a
"buildscript" section) and it looks like the lint plugin is very similar.
I'll experiment with getting it to work that way.
- There is a lot of redundancy in the deployment scripts for each module, and
given that we have the full power of a general programming language it
should be possible for us to reuse the common parts.
But that said, this is a pretty good first step and I wanted to put it out
there.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223366370
I'm adding another EPP lifecycle test that will need to be in the tools package
because it has to call tools as part of the lifecycle. This commit consolidates
common functionality within the EppTestCase abstract base class (and increases
visibility) so that it can easily be referenced by more extending classes, even
ones in a different package.
This also explicitly loads the test files from the testdata directory collocated
with EppTestCase, so that new tests in other packages won't have to duplicate
lots of these same test files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223365398
content-type needs to be parsed (no automatic parsing from String)
data was splitting on commas, meaning --data="key=value1,value2" was sent to the server as "key=value1&value2"
NOTE - you'd expect there to already be a "do nothing splitter", right? But there isn't :/
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223346887
EppResource's lastEppUpdateTime and lastEppUpdateClientId need
to be updated whenever the domain is updated, renewed, deleted or
transfered.
This commit applied the change to the following domain EPP commands:
- Update (already implemented)
- Renew
- Delete
- Restore
- Transfer request
- Transfer approve
- Transfer reject
- Transfer cancel
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223344758
Registrars that are PENDING or SUSPENDED should not have access to
EPP commands that cost money, as in either case it's not likely we'd
actually be able to get payment from said registrar. For this reason
we already prevented access to the domain create flow for non-active
registrars. This commit extends that to other commands that cost
money, including renewals, restores, and transfer requests.
Note that implicit autorenews will still occur for suspended
registrars, as in our point-in-time data model there's no good way
to prevent them. So when a registrar is suspended for non-payment,
the game plan is to get all of their domains transferred out to a
registrar that will pay as soon as possible.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223173677
As previously written, it loaded up all history entries into memory and then
processed them. This was OOMing for some registrars on sandbox who had performed
a large number of testing actions, most of them long OT&E was passed.
This commit changes the verify OT&E action to stream the history entries in
batches, ordered by when they were made, and then terminates once all tests have
passed. This prevents OOMing because only a single batch of history entries need
reside in memory at once.
This does necessitate the creation of a new composite Datastore index on
HistoryEntry, so we'll need to run the ResaveAllHistoryEntriesAction in sandbox
after this change is deployed before the new verify OT&E code will work.
Note that the "history viewer" is long dead, but that the pre-existing index
on HistoryEntries is still used for many other purposes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223163337
This allows us to provide the keyring a blob of encrypted data and a key name, and have it decrypt it for us.
Also fixed javadoc length in Keyring.java. It seems like it was using a 80-character length limit.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222995542
These targets are removed in [] as they are not compile-time
dependencies. But they are needed at runtime by GetRoutingMapCommand to load
the specific classes to generate routing map for.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222916432
This makes it possible to request the encrypted data directly in application code. It will be used to download service account credential during "nomulus login".
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222847905
The @JsonPayload qualifier is not used because the field are contrustor injected.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222829281
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
The console will have 2 different "updatable things":
- only ADMINs (GAE-admins and users in the support G-Suite group) can change the things in the "admin settings" tab (currently just the allowed TLDs)
- only OWNERs can change things from the other tabs: WHOIS info, certificates, whitelisted IPs, contacts etc.
Also, all ADMINs are now OWNERS of "non-REAL" registrars. Meaning - we're only
preventing ADMINs from editing "REAL" registrars (usually in production).
Specifically, OTE registrars on sandbox are NOT "REAL", meaning ADMINS will
still be able to update them.
This only changes the backend (registrar-settings endpoint). As-is, the console
website will still make ADMINs *think* they can change everything, but if they
try - they will get an error.
Changing the frontend will happen in the next CL - because I want to get this
out this release cycle and getting JS reviewed takes a long time :(
TESTED=deployed to alpha, and saw I can't update fields even as admin on REAL
registrars, but could change it on non-REAL registrars. Also checked that I can
update the allowed TLDs on REAL registrars
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222698270
EppConsoleAction still "manually" checks access by going over the
RegistrarContacts. We need it to use AuthenticatedRegistrarAccessor just like
every other part of the registrar console.
We still need to remove the (now unneeded) login EPP sent by the console, but that's left for a followup CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222404208
