We need this as a precursor to running refreshDnsForAllDomains to change our
TTLs from 3 minutes to an hour (see the associated bug for details).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=246011747
The main thrust of this is to create a common POJO that contains email content in a simple way, then have one class that converts that to an email and sends it. Any class that uses email should only have to deal with creating that POJO.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237883643
The redacted text for the email field displays a longer prompt to
contact the registrar, per the request filed at b/123573370.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232716133
This code was never finished or fully working anyway. It would require
substantial reworking for the Registry 3.0 migration because it's closely tied
to the Datastore model and App Engine MapReduce framework, both of which will be
going away. We can bring back some of these deleted test files as necessary
if/when we rewrite RDE import for the new schema.
On the plus side, in a relational database, RDE import will be much simpler.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231265578
There was no reason to have several different modules all providing a single
thing. This approach, which creates a single UtilsModule for everything in the
util package, is cleaner. This also removes provisioning of Random and
StringGenerator objects in RegistryConfig.ConfigModule, which don't belong
there because they aren't configuration options.
This also removes insecure random entirely; it was only used in a
single place to generate 24 bytes a couple times per day. We can live with the
lower speed if it means we don't have to worry about multiple types of Random,
or possibly using an insecure random accidentally in a place that security
actually does matter.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229751915
This console is only to be used by Admins (either GAE admins for this project, or Support accounts). It is for "internal" use only, not for use by the registrars themselves.
To prevent abuse, the registrar is created in a non-functional PENDING state and can only be made functional from the nomulus shell tool.
While in "PENDING" state, the registrar can be updated from the registrar-console by admins.
Also - moving all the web consoles to the same directory (moving the otesetup/* files into registrar/)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229681011
For each registrar, the daily email will only include threats that did not appear
in the prior run's email.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228889972
Certain flows need to launch batched jobs. Logically this would mean that flows
depend on batch.
However, the current state of dependency was the other way around, and the
reason for that was ResourceFlowUtils.java that had in it some utility
functions that weren't used in the flows and were needed in the batch jobs.
This CL removes these utility functions from the /flows/ directory, letting us
reverse the dependency edge between flows/ and batch/
Part of this was moving the flows/async/ code into batch/ - which also makes sense because flows/async/ just "enqueued" tasks that would then be run by actions in batch/
It makes sense that the code that enqueues the tasks and the code that dequeues the tasks sit in the same library.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228698761
SendEmailUtils is a general utility of the web console, and not specifically "only"
to the Registrar console.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226187094
In addition to just making good sense to not have support group for some
environments (local? unittest? crash?) - connecting with G Suit requires
additional permissions that are harder to find.
Specifically, it requires the Json Credentials that just aren't set in the
Dummy Keyring used on some environments.
So we make sure to not even *try* to create the credentials if the support
email isn't set
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225589255
Modularize the code for DNS count reporting to allow it to be customized for
more flexible systems.
Tested:
Uploaded to alpha with hacks to allow admin initiating and logging from the
DnsCountQueryCoordinatorModule, verified that the provider function is invoked and
that the action runs successfully.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225225587
Note that it's possible to set a config option to disable this functionality
on a per-environment basis (we're disabling it for sandbox), but in general
SSL certificate hashes should be required for increased security.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225053496
Add server end points to backup Datastore using managed-export mechanism.
A cron job is defined in Alpha to run daily exports using this implementation.
Existing backup is left running. The new backups are saved to a new set of
locations:
- GCS bucket: gs://PROJECT-ID-datastore-backups
- Big Query data set: datastore_backups
- Big Query latest back up view name: latest_datastore_backup
Also, the names of Bigquery tables now use the export timestamp
assigned by Datastore. E.g., 2018_12_05T23_56_18_50532_ContactResource,
After the new import mechanism is implemented and the back-restore flow is
tested, we will stop the existing backup runs and deploy the new
implementation to all environments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224932957
The scenarios in which the credential is used are:
1) Calls to Nomulus GAE HTTP endpoints.
2) Calls to Google APIs within the tool.
3) Calls to GAE APIs within the tool.
From now on the tool should not depend on ADCs created from gcloud any more (expect for beam pipeline deployments which need some more investigation as the dependency on ADC is not apparent). Using the nomulus tool requires running "nomulus login" first, but only once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224165735
RemoteApiOption has a package-private method that takes a Stream representing the content of a JSON and use a GoogleCredential created from it as its credential. This CL uses reflection to change the access modifier of that method in order to supply a credential stream that is self-managed. This is obviously not ideal and prone to breakage in case the getGoogleCredentialStream method is changed. Unfortunately upstream is not willing to make it public citing the reason that GoogleCredential.fromStream() (which getGoogleCredentialStream uses) is a @Beta annotated function (see https://groups.google.com[]forum/#!searchin/domain-registry-eng/remoteapioptions%7Csort:date/domain-registry-eng/Flsah6skszQ/CySZv2XEBwAJ). However this function is introduced 5 years ago as a public function (b857184bfa). I think at this point it is safe to assume that it is part of the widely used APIs and will not change without sufficient notice.
Note here that RemoteApiOptions creates its own copy of GoogleCredential to be used to call App Engine APIs locally, whereas communications to Nomulus endpoints use the Credential provided in AuthModule. Even though both credentials are created from the same client id, client secret and refresh token (the three elements needed to construct a GoogleCredential this way, see https://github.com/googleapis/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/GoogleCredential.java#L842), their refreshes cycles are independent of each other. I verified that refreshing one of the credential does not invalidate the access token of the other credential, as long as it is not expired yet.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224156131
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
This makes it simpler to package google.registry.util as a separate project in
Gradle that can be depended upon by the proxy package. Currently the proxy
package depends on both google.registry.util and google.registry.config.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221450085
After this CL, "support" accounts (accounts that are part of the "support" G-Suite group) will the same access to the registrar console as GCP "admins". However, they don't won't have access to the GCP project itself.
We could give them their own Role in the future (say SUPPORT) and give them different access than "admins", but right now we don't need it and YAGNI or something :)
NOTE: we identify users by their email (they need to be logged in to a google account). I don't know if that's best practice, since I guess different google accounts might have the same email address. However, G-Suite groups' membership is by email so there's not much we can do about it if we want to use G-Suite groups.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220804273
It was failing to send alert emails because the email address it was
constructing did not have permission through GAE to send emails. This switches
it over to using the send from email address already in use elsewhere in the app
that does successfully send emails.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219812019
AppEngineConnection can now connect to all services and not just the tools.
The default is still the tools.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218734983
Make every dependency request explicit on what encoding is used. Also get rid of InjectRule in XjcToDomainResourceConverterTest.
Random number generator providers are separated to secure and insecure ones. The insecure ones must be explicitly requested (usually for use cases where security is not of concern, for better speed).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217921422
Someone may @Inject a Random at some point in the future where security matters,
so always provide a secure one.
Also, this shouldn't be in ConfigModule (it's not configuration) -- but that can
be changed separately. We might want a larger refactor that has a utils module
or similar to provide extremely generic things like random number generators.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217602486
com.google.common.base.Randoms.insecureRandom is not open sourced.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217598084
Random used to be a static variable which requires InjectRule to mock it in unit tests. It is now a singleton, which ensures that the same instance is called every time and Random.nextBytes() generates results that distribute uniformly between each call.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217592767
This uses a Dagger-provided map of Keyring implementations, with two currently available,
"KMS" and "Dummy". The active keyring is configured in the YAML file, so we no longer
require MOE directives to choose which one to use for internal/external builds.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=216898058
Looks like a copy-paste error from the reserved list export disclaimer.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=214773560
Scope changes in delegated credentials require coordinated external changes,
therefore should be separate from those used in the application default
credential.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212488389
There's no real standard for commented lines in a CSV, but this seems to be the
most well-supported option, so may as well use it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211847395
As the first step in credential consolidation, we replace
injection of application default credential in for KMS and
Drive.
Tests:
- for Drive, tested with exportDomainLists and exportReservedTerms.
- For KMS, used CLI commands (get_keyring_secret and update_kms_keyring) to change and
restore secret for one key.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211819859
This adds the terminal step of the Spec11 pipeline- processing the output of
the Beam pipeline to send an e-mail to each registrar informing them of
identified 'bad urls.'
This also factors out methods common between invoicing (which uses similar beam pipeline tools) and spec11 to the common superpackage ReportingModule + ReportingUtils classes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=210932496
This adds actual subdomain verification via the SafeBrowsing API to the Spec11
pipeline, as well as on-the-fly KMS decryption via the GenerateSpec11Action to
securely store our API key in source code.
Testing the interaction becomes difficult due to serialization requirements, and will be significantly expanded in the next cl. For now, it verifies basic end-to-end pipeline behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208092942
All the pipeline-crashing problems should be fixed now, so we should have no
problem re-automating the invoice publish.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207265990
This adds the scaffolding for a basic Spec11 pipeline- it gathers all domains from all time for a given project and counts how many there are. I've factored out a few common utilities for beam pipelines to avoid excessive duplication.
Future CLs will:
- Actually process domains via the SafeBrowsing API
- Generate a real spec11 report
- Template queries based on the input YearMonth
- Abstract more commonalities across beam pipelines to reduce boilerplate when adding new pipelines.
TESTED: FOSS test passed, and ran successfully on alpha
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205997741
This prepares for the spec11 beam pipeline to live parallel to the invoicing
beam pipeline, for better organization.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204980582
Second step of RDE encoding refactoring.
Creates a single OutputStream encode RyDE files.
This replaces the 5 OutputStreams that were needed before.
Also removes all the factories that were injected. It's an encoding, there's no point in injecting it.
Finally, removed the buffer-size configuration and replaced with a static final
const value in each individual OutputStream.
This doesn't yet include a decoder (InputStream). And there's still a lot of overlap between the Ryde and the Ghostryde code. Both of those are left for the next CLs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204898369
We never launched this, don't planning on launching it now anyway, and it's rotted over the past two years anyway.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202993577
First step of RDE encoding refactoring.
Creates a single InputStream (OutputStream) to decode (encode) Ghostryde files.
This replaces the 3 InputStreams (OutputStreams) that were needed before.
Also removes a lot of classes, and removes the "injection" of the Ghostryde
class. It's an encoding, there's no point in injecting it.
Finally, removed the buffer-size configuration and replaced with a static final
const value. It's just a buffer size - it doesn't actually affect much. There
are much more "important" fields that weren't configured (such as the
compression algorithm and whether or not to do integrity checks)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202319102
Copied class and test from CheckApiAction. All unit tests passing.
Remaining work: add metrics
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198916177
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
This adds a parameter to control invoice reporting, which defaults to false for
now (since we plan on manually adjusting the invoice next month).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193708323
These servlets are converted to actions during daggerization. Calling them servers are misleading.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190942237
