Commit graph

456 commits

Author SHA1 Message Date
jianglai
abc508712d Restore RemoteApiOptions method accessibility after use
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=246161567
2019-05-06 16:24:44 -04:00
mcilwain
b46a6b6d55 Fix some statically detected code issues
This includes: unnecessary semicolons, suppress warnings, switch statements, final/private qualifiers, Optional wrapping, conditionals, both inline and non-inline variables, ternaries, Collection putAll() calls, StringBuilders, and throws declarations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=244182539
2019-04-22 12:54:34 -04:00
mcilwain
ff3aeff0ed Remove per-domain nameserver restrictions in reserved lists
This also removes the related setting on the Registry entity. We never used
either of these, and keeping them around in our codebase just adds complexity
for no purpose. We already achieve the goals of this feature by restricting
nameservers on entire TLDs and by registry-locking important domain names.

This is also two fewer things we'll have to worry about carrying over to the new
schema in Registry 3.0.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=243816241
2019-04-16 14:09:27 -04:00
gbrodman
74971144d2 Don't delete or "redeem" unlimited use AllocationTokens
We haven't started dealing with timing or discounts yet, but unlimited use tokens should actually be unlimited use

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=243318266
2019-04-15 14:57:02 -04:00
gbrodman
cfee7e7fd5 Add necessary fields to the AllocationToken schema
See https://docs.google.com/document/d/1SSWrILRpx0Mtr4sdvlYwz9I8wJp5Gu_o4qlml3iJDKI

This is just the base for now--we don't actually do anything with it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=243265164
2019-04-15 14:53:45 -04:00
shicong
5e310f8fcd Fix Bazel build
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=242472456
2019-04-11 14:47:31 -04:00
shicong
72b9ca6894 Add required dependency for RemoteApiInstaller
We got 500 Internal Server Error when calling GAE RemoteApi in Nomulus
tool. The root cause is we missed the dependency described in this
[]

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=242166362
2019-04-11 14:43:46 -04:00
mcilwain
5ef4c4edf1 Display created domain name in get_allocation_token command
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=240212479
2019-03-29 16:02:48 -04:00
mcilwain
91febfd945 Add a nomulus tool command to get allocation tokens
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239624232
2019-03-21 15:05:12 -04:00
mcilwain
d7306652eb Allow registrars to be completely DISABLED
Disabled registrar cannot perform any actions via EPP and cannot log in to the
registrar web console.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239606389
2019-03-21 15:03:33 -04:00
gbrodman
4544aa1efe Enforce abuse WHOIS contact for REAL registrars when adding TLDs
We do not enforce this for non-REAL registrars or in any environment other than UNITTEST or PRODUCTION. This is similar but separate to [] since we can add allowed TLDs in either location.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239510275
2019-03-21 15:00:16 -04:00
gbrodman
36fd13f8e0 Fall back to icannReferralEmail when creating registrars through the command line
This is if the separate email field isn't specified.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239504636
2019-03-21 14:56:53 -04:00
mcilwain
067756722d Change nomulus registrar --drive_id parameter to --drive_folder_id
This makes it consistent with the parameter of the same name on the tld commands.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=234148699
2019-02-18 17:32:57 -05:00
mcilwain
29d3ad8052 Switch to constructor injection in a few actions
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232552406
2019-02-06 17:53:49 -05:00
gbrodman
058ef07e16 Fix some style issues that will help us out later
These files will have errors later when we run the Google Java Format plugin over their entirety (e.g. a situation where fixed indentation leads to a line that's longer than 100 characters). It's simpler to fix them now so we won't have to fix them later.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232353791
2019-02-06 17:47:37 -05:00
mcilwain
339202e879 Delete the mapreduce poll action
We never used it for anything.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232024489
2019-02-01 16:24:37 -05:00
mcilwain
f0c677b18b Rename DNL and SMDRL "login" to "loginAndPassword"
They are passed around in the format username:password, whereas just saying
"login" implies it's just a username and not necessarily also a secret
password. Putting password in the variable name makes it obvious what this is
and reduces the likelihood of anyone ever logging it or otherwise using it
inappropriately.

Note that this does not require data migrations as the actual key used to store
the data in KMS remains unchanged.

This is a follow-up to []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231253964
2019-01-28 16:15:04 -05:00
mcilwain
c6e58d3bff Fix some issues caught by IntelliJ static code analysis
The most common issues were:
* Arrays.asList() shouldn't be called with a single parameter.
* Broken Javadoc @links.
* Unnecessary casts and type declarations.
* Unnecessary unused variable initializations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230994311
2019-01-28 16:08:24 -05:00
gbrodman
3cf26ff9b6 Fix various Error Prone errors that were found by the FOSS build
Most common:
- Unnecessary parentheses and operator precedence clarify (self-explanatory)
- Reference equality--there were a few instances of using == or != improperly
- Qualification of Builder (and similar) imports so that it's clear which type of Builder we're referring to
- Marking some immutable classes with @Immutable since EP desires that all enums be deeply immutable
- String.split() having "surprising behavior"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230971531
2019-01-28 16:05:09 -05:00
mcilwain
5b0c61dad3 Use simpler Iterables helper when counting domains
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230955066
2019-01-28 16:01:55 -05:00
mcilwain
e2528875b2 Merge DomainResource into DomainBase
This eliminates the use of Objectify polymorphism for EPP resources entirely
(yay!), which makes the Registry 3.0 database migration easier.

It is unfortunate that the naming parallelism of EppResources is lost between
ContactResource, HostResource, and DomainResource, but the actual type as far as
Datastore was concerned was DomainBase all along, and it would be a much more
substantial data migration to allow us to continue using the class name
DomainResource now that we're no longer using Objectify polymorphism. This
simply isn't worth it.

This also removes the polymorphic Datastore indexes (which will no longer
function as of this change). The non-polymorphic replacement indexes were added
in []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230930546
2019-01-28 15:57:10 -05:00
mcilwain
97c2049669 Catch errors when attempting to delete entities through Objectify
This is currently erroring out on entities that fail to load properly through Objectify (e.g. because their entity type is no longer registered).  The proper thing to do is to catch the error, log it, and fall back to the raw Datastore operation, which will succeed.

The exact Exception this is designed to catch is:

com.google.apphosting.runtime.jetty9.JettyLogger warn: /_dr/admin/deleteEntity (JettyLogger.java:29)
java.lang.IllegalStateException: No registered subclass for discriminator 'DomainApplication'
	at com.googlecode.objectify.v4.impl.PolymorphicEntityMetadata.getConcrete(PolymorphicEntityMetadata.java:133)
	at com.googlecode.objectify.v4.impl.PolymorphicEntityMetadata.load(PolymorphicEntityMetadata.java:164)
	at com.googlecode.objectify.v4.impl.LoadEngine.load(LoadEngine.java:220)
	at com.googlecode.objectify.v4.impl.LoadEngine$1.nowUncached(LoadEngine.java:178)
	at com.googlecode.objectify.v4.impl.LoadEngine$1.nowUncached(LoadEngine.java:164)
	at com.googlecode.objectify.v4.util.ResultCache.now(ResultCache.java:30)
	at com.googlecode.objectify.v4.impl.Round$1.nowUncached(Round.java:73)
	at com.googlecode.objectify.v4.util.ResultCache.now(ResultCache.java:30)
	at com.googlecode.objectify.v4.LoadResult.now(LoadResult.java:25)
	at google.registry.tools.server.DeleteEntityAction.loadOfyEntity(DeleteEntityAction.java:103)
	at google.registry.tools.server.DeleteEntityAction.run(DeleteEntityAction.java:74)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230737553
2019-01-24 20:02:04 -05:00
weiminyu
acbd23fa64 Remove deprecated Datastore backup code
Removed three Action classes and the CheckSnapshot command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230545631
2019-01-24 19:58:54 -05:00
gbrodman
701ebc6a28 Fix a few stylistic issues in preparation for Checkstyle addition
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230524735
2019-01-24 19:57:14 -05:00
mcilwain
d2ee63cf69 Consolidate Dagger modules for utils classes
There was no reason to have several different modules all providing a single
thing. This approach, which creates a single UtilsModule for everything in the
util package, is cleaner. This also removes provisioning of Random and
StringGenerator objects in RegistryConfig.ConfigModule, which don't belong
there because they aren't configuration options.

This also removes insecure random entirely; it was only used in a
single place to generate 24 bytes a couple times per day. We can live with the
lower speed if it means we don't have to worry about multiple types of Random,
or possibly using an insecure random accidentally in a place that security
actually does matter.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229751915
2019-01-17 19:20:52 -05:00
mcilwain
37aa1d1815 Always require acknowledgment of premium fees
This removes the configuration ability on both Registry and Registrar entities
to allow operations on premium domains to succeed without acking the fees using
the fee extension. We only ever used this ability during the minna launch, and
it was a fiasco. We have no intention of ever allowing creation, renewal,
transfer, restoring, etc. of premium domains without acking the fees ever again,
and haven't done so since 2013, so removing this ability allows us to simplify
our code, data model, and tests.

Note that all TLDs in our production system currently require price ACKing
anyway, so from an external partner perspective this commit is a noop.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229423650
2019-01-17 19:07:51 -05:00
guyben
a4f85c33c0 Add the App Engine service used in the Action definition
Our goal is to be able to address every Action by looking at the class itself, and to make it clearer at a glance what you need to access the Action's endpoint

Currently, we can know from the @Action annotation:
- the endpoint path
- the Method needed
- the authentication level needed

This CL adds the service where the Action is hosted, which also translates to the URL.

NOTE - currently we don't have any Action hosted on multiple services. I don't think we will ever need it (since they do the same thing no matter which service they are on, so why host it twice?), but if we do we'll have to update the code to allow it.

The next step after this is to make sure all the @Parameters are defined on the Action itself, and then we will be able to craft access to the endpoint programatically (or at least verify at run-time we crafted a correct URL)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229375735
2019-01-17 18:59:16 -05:00
weiminyu
9014165148 Add a Nomulus command to list Datastore operations
Command would be useful when checking the status of daily
backup jobs.

Only two modes are supported: list all operations, or list those
that started within the past period as specified by command line
option.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229239783
2019-01-14 16:31:18 -05:00
mcilwain
6082addb86 Make some quality of life improvements to set_num_instances command
Allows correct service names (i.e. lowercased, as they appear in App Engine
configuration files and the GCP console), and adds single letter parameters for
common flags.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229194479
2019-01-14 16:29:45 -05:00
mcilwain
8ac8ecf8f6 Rationalize OT&E client ID validation rules
This makes the validation rules much simpler, thus placing less cognitive load on the users of the console and nomulus tool.  The changes are:

1. Don't allow hyphens. No real registrars use hyphens in their client IDs, and it's better to reserve these solely as the delimiter between the base client ID and the number representing the environment.
2. Allow the first character to be a number.  This has affected multiple real registrars, causing their OT&E and production client IDs to be different.  There's no reason for this restriction, as the only reason motivating it was that there are no TLDs that start with a number.  However, the OT&E TLDs are created only in sandbox and never have DNS syncing enabled, so this restriction is purposeless.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229187198
2019-01-14 16:28:12 -05:00
guyben
fd8a18b72e Clean up streams in SetNumInstancesCommand
Also using Service instead of serviceId String - because it helps readability.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228952033
2019-01-14 16:26:22 -05:00
weiminyu
a80a44cd06 Define TestRule that manages cache use in tests
All current tests that use caches with custom data expiry values
now restore the default config when teardown. We need to prevent
new unsafe uses from being introduced.

Restoration code have also been added to a few other tests that modifies
static fields.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228888041
2019-01-11 11:50:33 -05:00
mcilwain
a6476862fd Use Guava instead of Apache Commons for file ops in Ghostryde
This backs out most of [] fixes the external build (which wasn't
finding Apache Commons correctly), and makes miscellaneous tweaks and fixes,
including better handling representing the default case of decrypting to stdout.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228877090
2019-01-11 11:00:33 -05:00
mcilwain
765e63e7e9 Send a plaintext link to the mapreduce console in fluent style
The link was previously being sent using a JS redirect, which doesn't work
because the endpoints that trigger mapreduces can only be hit from the command
line (because they require auth). This commit switches the link to be in
plaintext and renders the full URL instead of just the path, so that clicking it
directly from the terminal works.

This also improves how these links are sent from callsites by using a fluent
style.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228764606
2019-01-10 17:14:06 -05:00
mcilwain
580302898d Delete end-date sunrise, landrush, and sunrush phases
This also deletes the associated commands and domain application specific
entities.

We haven't used any of these TLD phases since early 2015 and have no
intent to do so in the future, so it makes sense to delete them now so we
don't have to carry them through the Registry 3.0 migration.

Note that, while there are data model changes, there should be no required
data migrations. The fields and entities being removed will simply remain
as orphans. I confirmed that the removed types (such as the SUNRUSH_ADD
GracePeriodType) are no longer used in production data, and left types
that are still used, e.g. BillingEvent.Flag.LANDRUSH or
HistoryEntry.Type.ALLOCATE.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228752843
2019-01-10 16:23:35 -05:00
guyben
c74ffd7559 Fix @VisibleForTesting given the newly deployed enforcement
Generated code is now also covered by @VisibleForTesting, including Dagger @Inject

This CL is a cleanup of auto-generated code by ghm@ from the Error Prone team

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228748874
2019-01-10 16:23:35 -05:00
mcilwain
d0c73efac0 Reduce logging level to warning for missing DAIs
This is expected as part of the normal way that this mapreduce runs, so it's not
worth outputting an error.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228553579
2019-01-10 16:23:35 -05:00
mcilwain
02174a2cff Delete applications even when index is missing
This makes the deletion mapreduce more resilient in the face of data integrity
violations (which exist on sandbox but hopefully not in production). Even when
the domain application index doesn't exist, we still want to delete the domain
application itself, as its continuing presence will cause problems after the
code for domain applications is deleted.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228521794
2019-01-10 16:23:35 -05:00
weiminyu
4e71421c81 Support datastore restore in Nomulus tool
Two commands are being added:
- ImportDatastoreCommand starts an async import operation.
  User may choose to wait until import completes or quit
  immediately.
- GetOperationStatusCommand checks the status of an operation.
  It may be used to check the status of an operation started by
  ImportDatastoreCommand.

Both commands communicate with Datastore admin api directly, without
going through the Registry server.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228400059
2019-01-10 16:23:35 -05:00
mmuller
3078efdaac Don't use Files.copy() when decrypting to stdout
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout.  Instead copy directly from the decoder
to standard output.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
2019-01-10 16:23:35 -05:00
mmuller
9e0b675275 Don't use Files.copy() when decrypting to stdout
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout.  Instead copy directly from the decoder
to standard output.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
2019-01-08 16:51:11 -05:00
guyben
9eaeab9cfe Allow query parameters in the connection's endpoint
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227898246
2019-01-08 10:53:35 -05:00
mcilwain
a81d45fe5d Fix stdout of DeleteAllocationTokensCommand
It was saying it was deleting tokens it wasn't, because it was outputting the
raw input list of tokens rather than the list that filtered out redeemed or
domain-specific tokens.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227769266
2019-01-08 10:51:55 -05:00
mcilwain
577c6f6bc9 Add a mapreduce to delete all domain applications
This also deletes associated entities including indexes and history entries.

This needs to run as a prerequisite to [] which deletes all domain
application code entirely. The entities themselves need to be deleted first so
that loading DomainBases in the future doesn't accidentally get applications
which the code no longer knows how to handle.

This deletion is safe to perform because the only remaining applications in our
system are historical and we no longer refer to them. Backups will be retained
in BigQuery.

This mapreduce will be deleted at the same time that the DomainApplication code
is.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227738528
2019-01-08 10:50:16 -05:00
guyben
566f60d495 Allow using empty string to remove all whitelisted IPs
Currently, you have to set "--ip_whitelist=null", which is unintuitive.

This adds the option to just give an empty string: "--ip_whitelist="

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227550896
2019-01-08 10:41:42 -05:00
jianglai
2e7d71b238 Refactor most of OT&E verification code to exist in utils class
This does not change existing functionality but will allow us to use this common code in the yet-to-be-created web console action as well.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=227159346
2019-01-02 11:58:36 -05:00
shicong
f83e96c448 Exit with error code when user is not logged in
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226390329
2018-12-21 15:55:08 -05:00
emcmanus
a153f8ec77 Switch remaining App Engine dependencies to built-at-head targets
Most dependencies on the Old SDK were switched in [] This is just catching up on some OAuth dependencies that remained and some remaining uses of Old build rules.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226337284
2018-12-21 15:55:08 -05:00
mmuller
2cc4a9fb2f Add MOE equivalence[s] for 2018-12-13 sync
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225418342
2018-12-21 15:55:08 -05:00
jianglai
27b6231053 Add the ability to provide credential JSON file to the nomulus tool
This allows us to run nomulus tool programmatically on environments that do not
allow the 3-legged OAuth authentication flow.

The provided JSON file corresponds to a service account, which must have
GAE admin permission and whose client ID must be whitelisted in the config
file.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226008337
2018-12-20 07:46:33 -05:00