Commit graph

112 commits

Author SHA1 Message Date
weiminyu
8a4407a9a1 Replace deprecated Mockito Matchers class
Part of the attempt to remove or suppress warnings for deprecated
API use, which will make the Gradle project usable with Intellij.

Currently in the Intellij/Gradle setup, deprecation warnings cause
Intellij build process to fail. Passing -Werror:none flags to javac\
does not have any effect.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=246135737
2019-05-06 16:21:27 -04:00
mcilwain
b46a6b6d55 Fix some statically detected code issues
This includes: unnecessary semicolons, suppress warnings, switch statements, final/private qualifiers, Optional wrapping, conditionals, both inline and non-inline variables, ternaries, Collection putAll() calls, StringBuilders, and throws declarations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=244182539
2019-04-22 12:54:34 -04:00
Ben McIlwain
351eba3f6d Allow third_party/java_src/gtld/ to use :mockito2_for_third_party
In [] a change would have been made to your project that is incompatible with
your open source integration. To make sure the open source variant of your
project remains working, we have eagerly updated your open source copy to use
Mockito 2. This CL integrates that change into [].

Please read []and
understand the consequences of this change.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=238445356
2019-03-20 14:25:28 -04:00
mcilwain
241dbea9d9 De-premiumize wing.dev for internal registration
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=236661149
2019-03-20 14:25:28 -04:00
jianglai
b573b10ae1 Remove backported MockitoJUnitRule
Mockito in third_party is updated to 1.10. We do not need to backport this rule anymore.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237496086
2019-03-08 18:35:00 -05:00
mcilwain
3812c2ceda Use -dot- subdomain notation in MapReduce console links
appspot.com is not provisioned with a multi-level wildcard SSL certificate, so
URLs of the form https://service.projectid.appspot.com/path need to be rewritten
as https://service-dot-projectid.appspot.com/path (and same for version names).

This is a follow-up to []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231418795
2019-01-31 13:59:05 -05:00
mcilwain
c6e58d3bff Fix some issues caught by IntelliJ static code analysis
The most common issues were:
* Arrays.asList() shouldn't be called with a single parameter.
* Broken Javadoc @links.
* Unnecessary casts and type declarations.
* Unnecessary unused variable initializations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230994311
2019-01-28 16:08:24 -05:00
weiminyu
acbd23fa64 Remove deprecated Datastore backup code
Removed three Action classes and the CheckSnapshot command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230545631
2019-01-24 19:58:54 -05:00
mcilwain
765e63e7e9 Send a plaintext link to the mapreduce console in fluent style
The link was previously being sent using a JS redirect, which doesn't work
because the endpoints that trigger mapreduces can only be hit from the command
line (because they require auth). This commit switches the link to be in
plaintext and renders the full URL instead of just the path, so that clicking it
directly from the terminal works.

This also improves how these links are sent from callsites by using a fluent
style.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228764606
2019-01-10 17:14:06 -05:00
mcilwain
580302898d Delete end-date sunrise, landrush, and sunrush phases
This also deletes the associated commands and domain application specific
entities.

We haven't used any of these TLD phases since early 2015 and have no
intent to do so in the future, so it makes sense to delete them now so we
don't have to carry them through the Registry 3.0 migration.

Note that, while there are data model changes, there should be no required
data migrations. The fields and entities being removed will simply remain
as orphans. I confirmed that the removed types (such as the SUNRUSH_ADD
GracePeriodType) are no longer used in production data, and left types
that are still used, e.g. BillingEvent.Flag.LANDRUSH or
HistoryEntry.Type.ALLOCATE.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228752843
2019-01-10 16:23:35 -05:00
weiminyu
4e71421c81 Support datastore restore in Nomulus tool
Two commands are being added:
- ImportDatastoreCommand starts an async import operation.
  User may choose to wait until import completes or quit
  immediately.
- GetOperationStatusCommand checks the status of an operation.
  It may be used to check the status of an operation started by
  ImportDatastoreCommand.

Both commands communicate with Datastore admin api directly, without
going through the Registry server.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228400059
2019-01-10 16:23:35 -05:00
weiminyu
9c706e79fd Backup Datastore using the Admin REST API
Add server end points to backup Datastore using managed-export mechanism.
A cron job is defined in Alpha to run daily exports using this implementation.

Existing backup is left running. The new backups are saved to a new set of
locations:
- GCS bucket: gs://PROJECT-ID-datastore-backups
- Big Query data set: datastore_backups
- Big Query latest back up view name: latest_datastore_backup
Also, the names of Bigquery tables now use the export timestamp
assigned by Datastore. E.g., 2018_12_05T23_56_18_50532_ContactResource,

After the new import mechanism is implemented and the back-restore flow is
tested, we will stop the existing backup runs and deploy the new
implementation to all environments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224932957
2018-12-12 13:22:34 -05:00
weiminyu
7d380256af Add helper methods to DatastoreAdmin Operation object
These are needed by Datastore export management actions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224242281
2018-12-06 13:30:56 -05:00
weiminyu
676fa422db Use proper external dependency references recognized by moe
This fixes kokoro build failure.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221830188
2018-11-16 17:03:50 -05:00
weiminyu
af21b0c32b Add a package to open source
Add {java,tests}/google/registry/export/datastore to open source.
This is part of the migration to Datastore Managed Import/Export
for backup.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221800709
2018-11-16 17:02:15 -05:00
weiminyu
961e5cc7c7 Use @DefaultCredential for Cloud API access in GAE
This change completes the switch to @DefaultCredential for
all use cases in GAE.

Impacted modules:
- IcannReporting
- CreateCdnsTld command
- LoadSnapshot command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=213511730
2018-09-20 11:19:36 -04:00
weiminyu
80b0e6297b Export Premium names to Drive
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212509587
2018-09-14 11:47:38 -04:00
mcilwain
72bfd43e00 Add octothorpes to disclaimer in exported reserved list
There's no real standard for commented lines in a CSV, but this seems to be the
most well-supported option, so may as well use it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211847395
2018-09-08 00:18:19 -04:00
weiminyu
9436ce6f0e Introduce simplified Default credential provision
As the first step in credential consolidation, we replace
injection of application default credential in for KMS and
Drive.

Tests:
- for Drive, tested with exportDomainLists and exportReservedTerms.
- For KMS, used CLI commands (get_keyring_secret and update_kms_keyring) to change and
  restore secret for one key.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211819859
2018-09-08 00:16:09 -04:00
mcilwain
7b87ba41c7 Delete all Limited Release Program (LRP) code
We never used it and don't have any plans to use it going forward. All
conceivable parts of its functionality that we might use going forward have
already been subsumed into allocation tokens, which are a simpler way of
handling the same use case that are also standards-compliant.

Also gets rid of the hideous ANCHOR_ prefix on anchor tenant EPP authcodes
that was only ever necessary because of overloading the authcode for
anchor tenant creation. Going forward it'll be based on allocation tokens,
so there's no risk of conflicts.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209418194
2018-08-20 14:28:16 -04:00
mcilwain
6e74ba0587 Wrap ModulesService in new AppEngineServiceUtils
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).

This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.

This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
2018-08-10 13:44:25 -04:00
mcilwain
a4e99b18e2 Clean up internal imports in BUILD files
This removes the following unnecessary imports:

//third_party/java_src/gtld/java/google/registry/bigquery
//third_party/java_src/gtld/java/google/registry/config
//third_party/java_src/gtld/java/google/registry/cron
//third_party/java_src/gtld/java/google/registry/dns
//third_party/java_src/gtld/java/google/registry/gcs
//third_party/java_src/gtld/java/google/registry/mapreduce
//third_party/java_src/gtld/java/google/registry/model
//third_party/java_src/gtld/java/google/registry/module/backend
//third_party/java_src/gtld/java/google/registry/module/frontend
//third_party/java_src/gtld/java/google/registry/module/pubapi
//third_party/java_src/gtld/java/google/registry/module/tools
//third_party/java_src/gtld/java/google/registry/request
//third_party/java_src/gtld/java/google/registry/security
//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers
//third_party/java_src/gtld/java/google/registry/util
//third_party/java_src/gtld/java/google/registry/xjc
//third_party/java_src/gtld/javatests/google/registry/model
//third_party/java_src/gtld/javatests/google/registry/testing
//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce

The exact command run to generate this CL was:

build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java_src/gtld/java/google/registry/bigquery,//third_party/java_src/gtld/java/google/registry/config,//third_party/java_src/gtld/java/google/registry/cron,//third_party/java_src/gtld/java/google/registry/dns,//third_party/java_src/gtld/java/google/registry/gcs,//third_party/java_src/gtld/java/google/registry/mapreduce,//third_party/java_src/gtld/java/google/registry/model,//third_party/java_src/gtld/java/google/registry/module/backend,//third_party/java_src/gtld/java/google/registry/module/frontend,//third_party/java_src/gtld/java/google/registry/module/pubapi,//third_party/java_src/gtld/java/google/registry/module/tools,//third_party/java_src/gtld/java/google/registry/request,//third_party/java_src/gtld/java/google/registry/security,//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers,//third_party/java_src/gtld/java/google/registry/util,//third_party/java_src/gtld/java/google/registry/xjc,//third_party/java_src/gtld/javatests/google/registry/model,//third_party/java_src/gtld/javatests/google/registry/testing,//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202652421
2018-07-14 01:37:03 -04:00
mcilwain
bb1b4bc0d4 Clean up third_party imports in BUILD files
This removes the following unnecessary imports:

//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8

The exact command run to generate this CL was:

build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
2018-07-14 01:11:51 -04:00
mcilwain
43ed2cd7b3 Clean up annotation imports in BUILD files
This affects JSR305, JSR330, and Guava annotations.

The exact command run to generate this CL was:

build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
2018-06-27 15:28:53 -04:00
mcilwain
bee3d6a5a4 Fix BUILD glob rules to not include non-wildcarded paths
This is a Buildozer warning

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202311866
2018-06-27 15:28:53 -04:00
mcilwain
4c7bc3b18c Improve internal build system speed
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202016862
2018-06-27 15:28:52 -04:00
mcilwain
6706b99828 Delete remnants of registrar-level credits
We never fully used this stuff but definitely no longer use it following our
recent billing refactor. It's confusing to retain all of these entities and
commands given that none of them are actually used by anything.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201978094
2018-06-27 15:28:52 -04:00
mcilwain
4cddbdacff Delete deprecated legacy SQL 'latest_snapshot' dataset
We've migrated everything over to using the standard SQL view now. The legacy
version is just causing confusion and costing us resources.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201966352
2018-06-27 15:28:52 -04:00
mcilwain
6f2e663b72 Add asynchronous scheduled actions to re-save entities
This is used in the domain transfer and delete flows, both of which are
asynchronous flows that have implicit default actions that will be taken at some
point in the future. This CL adds scheduled re-saves to take place soon after
those default actions would become effective, so that they can be re-saved
quickly if so.

Unfortunately the redemption grace period on our TLDs is 35 days, which exceeds
the 30 day maximum task ETA in App Engine, so these won't actually fire.  That's
fine though; the deletion is actually effective as of 5 days, and this is just
removing the grace period.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201345274
2018-06-27 15:28:52 -04:00
mcilwain
ad73f3d167 Remove more unnecessary "throws" declarations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201243722
2018-06-27 15:28:52 -04:00
mcilwain
8b263baefa Delete MultiplyingCloudDnsWriter
Now that the large zone re-signing test is complete, we no longer need it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
2018-06-27 15:28:06 -04:00
mcilwain
5d80f124ca Remove unnecessary "throws" declarations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201058582
2018-06-18 18:17:56 -04:00
guyben
646c6fa069 Print GAE-UserId even when null
Also explicitly state that contacts missing GAE-UserId can't access the
registrar console

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200402953
2018-06-18 18:06:23 -04:00
jianglai
c0a7bde95e Remove deprecated PublishDetailReportAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198624767
2018-06-06 14:59:30 -04:00
jianglai
70b13596e4 Migrate to Flogger (green)
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.

No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.

Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.

For more information, see []
Base CL: 197826149

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
2018-05-30 12:18:54 -04:00
jianglai
1f1705aaa6 Migrate to flogger (red)
This is a 'red' Flogger migration CL. Red CLs contain changes which are
likely not to work without manual intervention.

Note that it may not even be possible to directly migrate the logger
usage in this CL to the Flogger API and some additional refactoring may
be required. If this is the case, please note that it should be safe to
submit any outstanding 'green' and 'yellow' CLs prior to tackling this.

If you feel that your use case is not covered by the existing Flogger API
please raise a feature request at []and
revert this CL.

For more information, see []
Base CL: 197826149

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198463651
2018-05-30 12:18:54 -04:00
mcilwain
c989911526 Batch NORDN pull queue task deletions
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.

Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
2018-05-17 21:52:35 -04:00
larryruili
6cdbde107f Redirect Registrar.referralUrl UI actions to url field
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196597051
2018-05-17 21:52:35 -04:00
jianglai
ebce333d5b Export registered domains to drive folder
The export happens in a reducer, whois instance fields all need to be serializable. The DriveConnection is therefore installed as a class variable that can be replaced with a mock during test. Class variables are not serialized. Only fields related to a particular instance are.

Note that DriveConnection is a misnomer. It is not a connection at all. It is just a thin wrapper class around the Drive service class, which provides convenient methods to write into Drive. Regardless, it cannot be serialized.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194596695
2018-05-05 23:38:42 -04:00
mcilwain
183dae6e80 Migrate away fully from MockitoJUnitRunner
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192291786
2018-04-10 17:01:04 -04:00
cushon
606b470cd0 Merge JUnitBackport's expectThrows into assertThrows
More information: https://github.com/junit-team/junit5/issues/531

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187034408
2018-03-06 18:56:15 -05:00
mcilwain
c416b3892d Migrate final try/catch test assertions to use assert/expectThrows
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182091814
2018-01-19 14:43:12 -05:00
mcilwain
f6f43badd4 Add basic AllocationToken entity and tests
Logic actually using this entity to follow in subsequent CLs introducing
the command to generate/save these entities as well as the flow logic for
considering them during domain EPP operations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181342600
2018-01-19 14:22:30 -05:00
mcilwain
ed0670b614 Remove unnecessary type specifications
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179585680
2017-12-27 11:23:03 -05:00
mcilwain
fbe11ff33c Use method references instead of lambdas when possible
The assertThrows/expectThrows refactoring script does not use method
references, apparently.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089048
2017-12-27 10:55:03 -05:00
mcilwain
7dc224627f Automatically refactor more exception testing to use new JUnit rules
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178911894
2017-12-27 10:42:36 -05:00
mcilwain
03c782f38e Replace ExceptionRule with ExpectedException
This is in preparation for running the automatic refactoring script that
will replace all ExpectedExceptions with use of JUnit 4.13's assertThrows/
expectThrows.

Note that I have recorded the callsites of assertions about EppExceptions
being marshallable and will edit those specific assertions back in after
running the automatic refactoring script (which do not understand these).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178812403
2017-12-13 12:43:45 -05:00
mcilwain
b825a2b5a8 Get rid of custom ExceptionRule methods
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.

Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
2017-12-13 12:43:45 -05:00
jianglai
1c1f95992a Move backported JUnit file to third_party (part 2)
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
2017-12-02 11:37:46 -05:00
mcilwain
e2db3f914e Clean up some code quality issues
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
2017-12-01 22:14:06 -05:00