Also using Service instead of serviceId String - because it helps readability.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228952033
Takes advantage of the fact that the default state of a TLD created in tests is GENERAL_AVAILABILITY.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228916164
For each registrar, the daily email will only include threats that did not appear
in the prior run's email.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228889972
All current tests that use caches with custom data expiry values
now restore the default config when teardown. We need to prevent
new unsafe uses from being introduced.
Restoration code have also been added to a few other tests that modifies
static fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228888041
This backs out most of [] fixes the external build (which wasn't
finding Apache Commons correctly), and makes miscellaneous tweaks and fixes,
including better handling representing the default case of decrypting to stdout.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228877090
The link was previously being sent using a JS redirect, which doesn't work
because the endpoints that trigger mapreduces can only be hit from the command
line (because they require auth). This commit switches the link to be in
plaintext and renders the full URL instead of just the path, so that clicking it
directly from the terminal works.
This also improves how these links are sent from callsites by using a fluent
style.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228764606
This also deletes the associated commands and domain application specific
entities.
We haven't used any of these TLD phases since early 2015 and have no
intent to do so in the future, so it makes sense to delete them now so we
don't have to carry them through the Registry 3.0 migration.
Note that, while there are data model changes, there should be no required
data migrations. The fields and entities being removed will simply remain
as orphans. I confirmed that the removed types (such as the SUNRUSH_ADD
GracePeriodType) are no longer used in production data, and left types
that are still used, e.g. BillingEvent.Flag.LANDRUSH or
HistoryEntry.Type.ALLOCATE.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228752843
Fix a couple more cases when caching related configs are changed.
This does not fix all our problems with Gradle builds.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228735790
Certain flows need to launch batched jobs. Logically this would mean that flows
depend on batch.
However, the current state of dependency was the other way around, and the
reason for that was ResourceFlowUtils.java that had in it some utility
functions that weren't used in the flows and were needed in the batch jobs.
This CL removes these utility functions from the /flows/ directory, letting us
reverse the dependency edge between flows/ and batch/
Part of this was moving the flows/async/ code into batch/ - which also makes sense because flows/async/ just "enqueued" tasks that would then be run by actions in batch/
It makes sense that the code that enqueues the tasks and the code that dequeues the tasks sit in the same library.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228698761
This makes the deletion mapreduce more resilient in the face of data integrity
violations (which exist on sandbox but hopefully not in production). Even when
the domain application index doesn't exist, we still want to delete the domain
application itself, as its continuing presence will cause problems after the
code for domain applications is deleted.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228521794
Two commands are being added:
- ImportDatastoreCommand starts an async import operation.
User may choose to wait until import completes or quit
immediately.
- GetOperationStatusCommand checks the status of an operation.
It may be used to check the status of an operation started by
ImportDatastoreCommand.
Both commands communicate with Datastore admin api directly, without
going through the Registry server.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228400059
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout. Instead copy directly from the decoder
to standard output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
There were two issues in the Gradle build.
1. Missing soyToJava conversion for newly added
google.registry.ui.soy.otesetup.
2. testSuccess_setAllowedTldsUncached_newTldNotInCache in
RegistrarTest modified the cache duration to 600 seconds
which broke other tests as those tests write to the storage
in initiation stage and expect the data to be available in
the cache immediately, this requires the duration to be
0.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228377715
It was saying it was deleting tokens it wasn't, because it was outputting the
raw input list of tokens rather than the list that filtered out redeemed or
domain-specific tokens.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227769266
This also deletes associated entities including indexes and history entries.
This needs to run as a prerequisite to [] which deletes all domain
application code entirely. The entities themselves need to be deleted first so
that loading DomainBases in the future doesn't accidentally get applications
which the code no longer knows how to handle.
This deletion is safe to perform because the only remaining applications in our
system are historical and we no longer refer to them. Backups will be retained
in BigQuery.
This mapreduce will be deleted at the same time that the DomainApplication code
is.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227738528
Currently, you have to set "--ip_whitelist=null", which is unintuitive.
This adds the option to just give an empty string: "--ip_whitelist="
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227550896
This does not change existing functionality but will allow us to use this common code in the yet-to-be-created web console action as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227159346
We create a new endpoint with a simple form that will let admins (including
support) setup OT&E for registrars.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226570568
SendEmailUtils is a general utility of the web console, and not specifically "only"
to the Registrar console.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226187094
This will only affect the tools service, the primary use case being (1) I go to
create a domain through nomulus tool, realize it's premium, (2) update the
premium list to not include that domain, (3) kill the tools service instance to
wipe out the cached premium value, then (4) create the domain at standard. This
commit eliminates step 3.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226180160
HTML headers can only be sent via JS, we need this change to allow
secure POST form submission.
The form itself will have a hidden "input" tag with the XSRF token in it. This
is how other framework do it as well - see
https://en.wikipedia.org/wiki/Cross-site_request_forgery#Synchronizer_token_pattern
This is in preparation for the OT&E setup page, which will be a simple form
with a "submit" button, so using JS for it is overkill.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226178070
This allows us to run nomulus tool programmatically on environments that do not
allow the 3-legged OAuth authentication flow.
The provided JSON file corresponds to a service account, which must have
GAE admin permission and whose client ID must be whitelisted in the config
file.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226008337
This is in preparation for having a web-console endpoint to create OTE.
In addition - we streamline the code:
- we remove support for different premium lists
- we remove support for different DNS writers - we never want a "real" DnsWriter for OTE
- we remove support of --eap_only, because we don't need it anymore
- We use a single password for all the Registrars
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225841694
In addition to just making good sense to not have support group for some
environments (local? unittest? crash?) - connecting with G Suit requires
additional permissions that are harder to find.
Specifically, it requires the Json Credentials that just aren't set in the
Dummy Keyring used on some environments.
So we make sure to not even *try* to create the credentials if the support
email isn't set
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225589255
This is safer and addresses a common source of confusion in the codebase because it's always explicit that the resource returned may not be present, whether because it's soft-deleted when projected to the given time or because it never existed in the first place.
In production code, the presence of the returned value is always checked. In test code, its presence is assumed using .get() where that is expected and convenient, as it not being present will throw an NPE that will cause the test to fail anyway.
Note that the roughly equivalent reloadResourceByForeignKey(), which is widely used in test code, is not having this same treatment applied to it. That is out of the scope of this CL, and has much smaller returns anyway because it's only used in tests (where the unexpected absence of a given resource would just cause the test to fail).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225424002
Modularize the code for DNS count reporting to allow it to be customized for
more flexible systems.
Tested:
Uploaded to alpha with hacks to allow admin initiating and logging from the
DnsCountQueryCoordinatorModule, verified that the provider function is invoked and
that the action runs successfully.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225225587
Tested:
TAP --sample for global presubmit queue
[] dr-foss-test passes
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225189790
Next up (and a much larger commit) will be giving loadByForeignKey() the same
treatment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225182377
Note that it's possible to set a config option to disable this functionality
on a per-environment basis (we're disabling it for sandbox), but in general
SSL certificate hashes should be required for increased security.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225053496
This will allow us to perform the OT&E history verification
in the model/ package as well so that it can be used both
by both the UI and the command line tool.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225007167
This is used to reduce the expiration time of domain(s) by some number of years
(if enough length remains in the registration term to do so). This does not back
out the previously saved BillingEvent entities as they may have already been
sent out and invoiced, so any related refunds must be handled out of band.
In addition to reducing the registration expiration time on the domain itself,
this command writes out a new history entry, one-time poll message informing the
registrar of this change, auto-renew billing event and poll message, and
updates/ends the old auto-renew billing event and poll message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224999285
Add server end points to backup Datastore using managed-export mechanism.
A cron job is defined in Alpha to run daily exports using this implementation.
Existing backup is left running. The new backups are saved to a new set of
locations:
- GCS bucket: gs://PROJECT-ID-datastore-backups
- Big Query data set: datastore_backups
- Big Query latest back up view name: latest_datastore_backup
Also, the names of Bigquery tables now use the export timestamp
assigned by Datastore. E.g., 2018_12_05T23_56_18_50532_ContactResource,
After the new import mechanism is implemented and the back-restore flow is
tested, we will stop the existing backup runs and deploy the new
implementation to all environments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224932957
Eventually the Publish action will control daily/monthly sending and provide
the correct threats to email. The goal of this PR is to entirely separate
the "sending email" functionality from the "parsing threat matches"
functionality.
The PublishAction will figure out if the monthly emails should be sent out,
then will ask the Spec11ThreatMatchesParser for the monthly threats (if
appropriate) and the new threat matches for today. It will then pass those
matches and the appropriate email subject+body to the email utils class,
whose only job is to format and send the emails.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224869643
The lastEppUpdateTime should be updated asynchronously in the situations below:
- Implicit transfer success after 5 day pending transfer period
- Implicit end of any grace period
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224831791
When a connection to the proxy using the PROXY protocol (https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) comes from an IP address that the external load balancer does not recognize, make the source IP 0.0.0.0. This way an appropriate WHOIS quota can be configured for this kind of connections.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224583547
This is for consistency, mostly the LocalDate fields added in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224525451
Add a sendSpec11Email parameter that allows us to only send the email on
one run per month. Next, we will compute the diffs between the daily runs
and send daily emails with those diffs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224404653
We are moving away from using Application Default Credentials generated by "gcloud auth application-default login" in our code base and consolidate on using self-managed credentials provided from AuthModule.
One of the remaining dependencies on the ADCs is from beam pipeline deployment commands, which by default use the ADCs to talk to GCS and upload the jar files and templates. In this CL, we explicitly provide the locally created credential to the Options used in deployments.
Also moved all credential qualifiers to CredentialModule, and removed @AppEngineAdminApiCredential, which is no longer used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224199812
The scenarios in which the credential is used are:
1) Calls to Nomulus GAE HTTP endpoints.
2) Calls to Google APIs within the tool.
3) Calls to GAE APIs within the tool.
From now on the tool should not depend on ADCs created from gcloud any more (expect for beam pipeline deployments which need some more investigation as the dependency on ADC is not apparent). Using the nomulus tool requires running "nomulus login" first, but only once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224165735
RemoteApiOption has a package-private method that takes a Stream representing the content of a JSON and use a GoogleCredential created from it as its credential. This CL uses reflection to change the access modifier of that method in order to supply a credential stream that is self-managed. This is obviously not ideal and prone to breakage in case the getGoogleCredentialStream method is changed. Unfortunately upstream is not willing to make it public citing the reason that GoogleCredential.fromStream() (which getGoogleCredentialStream uses) is a @Beta annotated function (see https://groups.google.com[]forum/#!searchin/domain-registry-eng/remoteapioptions%7Csort:date/domain-registry-eng/Flsah6skszQ/CySZv2XEBwAJ). However this function is introduced 5 years ago as a public function (b857184bfa). I think at this point it is safe to assume that it is part of the widely used APIs and will not change without sufficient notice.
Note here that RemoteApiOptions creates its own copy of GoogleCredential to be used to call App Engine APIs locally, whereas communications to Nomulus endpoints use the Credential provided in AuthModule. Even though both credentials are created from the same client id, client secret and refresh token (the three elements needed to construct a GoogleCredential this way, see https://github.com/googleapis/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/GoogleCredential.java#L842), their refreshes cycles are independent of each other. I verified that refreshing one of the credential does not invalidate the access token of the other credential, as long as it is not expired yet.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224156131
