Commit graph

2714 commits

Author SHA1 Message Date
mcilwain
6a6c7e3b09 Explain SERVER_HOLD part of NAME_COLLISION reservations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232713573
2019-02-06 18:01:37 -05:00
mmuller
a608e9f795 Check in initial dependency lockfiles.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232692353
2019-02-06 18:00:02 -05:00
mcilwain
29d3ad8052 Switch to constructor injection in a few actions
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232552406
2019-02-06 17:53:49 -05:00
guyben
927e8bbd73 Move LocalDate injection to the Actions themselves
We want to make it clear what query (or POST) inputs the user needs to / can give for each Action. That means moving all the @Injects of these parameters to the Actions themselves instead of injecting them in "hidden" indirect dependencies.

This has the extra benefit of allowing these indirect dependencies to work for JSON Actions as well, since the "regular" way we @Inject parameters can corrupt the POST JSON data.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232540758
2019-02-06 17:52:16 -05:00
gbrodman
89802329b3 Run more formatting fixes in preparation for the open-source change
Similar to [] these are issues found by Google Java Format. Most of the output is just using the standard [] formatter, then fixing any line-length issues.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232488284
2019-02-06 17:49:11 -05:00
gbrodman
058ef07e16 Fix some style issues that will help us out later
These files will have errors later when we run the Google Java Format plugin over their entirety (e.g. a situation where fixed indentation leads to a line that's longer than 100 characters). It's simpler to fix them now so we won't have to fix them later.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232353791
2019-02-06 17:47:37 -05:00
weiminyu
a325f0a50b Use Gradle native dependency-locking mechanism
Remove Nebula plugin and enable Gradle locking.

Update publishing task to include build dependencies in resolved artifacts
list.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232349134
2019-02-06 17:46:04 -05:00
mcilwain
339202e879 Delete the mapreduce poll action
We never used it for anything.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232024489
2019-02-01 16:24:37 -05:00
mmuller
7596a0b773 Delete unused testdata files
These files are unused by the tests in this directory, have no obvious purpose
and have no README explaining their existence.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232014732
2019-02-01 16:23:05 -05:00
mmuller
effbf836fd Add BaseDnsWriterTest to opensource tree
This should have gone in with the implementation of BaseDnsWriter.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232009971
2019-02-01 16:21:32 -05:00
mcilwain
f01ded3fa7 Make logged response in NordnUploadAction intelligible
Right now it's logging the raw bytes, which look like:

response data: [65, 117, 116, 104, 111, 114, 105, 122, 97, 116, 105, 111, 110, 32, 114, 101, 113, 117, 105, 114, 101, 100]

We'd rather convert it to ASCII characters (what the NORDN service uses) before
logging it, so that it'd instead look like:

response data: Authorization required

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231998658
2019-02-01 16:20:01 -05:00
guyben
c5ad30f49d Prevent spellchecking from textarea fields
The spellchecking causes test flakiness.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231984097
2019-02-01 16:18:24 -05:00
weiminyu
8085468280 Lock transitive dependencies in Gradle project
By default, only direct dependencies are locked.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231821158
2019-01-31 14:34:30 -05:00
weiminyu
8115909099 Add deploy notes to Gradle README
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231612510
2019-01-31 14:03:45 -05:00
mmuller
96f0c519f6 Move update_dependency.sh to g3/d/r/tools
We don't need this script to be publicly visible.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231606840
2019-01-31 14:02:15 -05:00
mcilwain
3812c2ceda Use -dot- subdomain notation in MapReduce console links
appspot.com is not provisioned with a multi-level wildcard SSL certificate, so
URLs of the form https://service.projectid.appspot.com/path need to be rewritten
as https://service-dot-projectid.appspot.com/path (and same for version names).

This is a follow-up to []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231418795
2019-01-31 13:59:05 -05:00
gbrodman
6e0b8f2cfd Pass the date field in to the Spec11 Soy template
The daily template is the only one that needs it but we can always pass it in without issue.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231295089
2019-01-31 13:57:30 -05:00
mcilwain
2edb072e68 Delete the deprecated RegistrarBillingEntry Datastore index
This should have been deleted in [] when the underlying entity was
deleted, but it was missed. It's been a no-op since then. This is just cleanup.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231293033
2019-01-31 13:55:52 -05:00
mcilwain
88aa13d1a4 Log a warning when MarksDB password isn't configured
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231287550
2019-01-31 13:51:10 -05:00
gbrodman
44e822ed86 Add Checkstyle and Error Prone to the Gradle build
We make some modifications to the internal Google checkstyle file because Google's linter uses a modified build of Checkstyle that introduces some new classes and allows for more specific checks than the open-source Checkstyle (e.g. only enforcing UPPER_SNAKE_CASE on deeply immutable fields). There exists a public Checkstyle file that purports to be the Google java format file (https://github.com/checkstyle/checkstyle/blob/master/src/main/resources/google_checks.xml) but it doesn't quite match up with what the internal linter says in certain situations (e.g. what operators must/can appear on new lines).

The suppressions are basically "don't run on generated code + don't care about Javadoc in test code"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231283700
2019-01-31 13:49:33 -05:00
Michael Muller
edc29d1a5a Adding png file to test image diffs in reviewable. 2019-01-31 12:40:26 -05:00
mcilwain
5dedc1e889 Delete everything related to RDE import
This code was never finished or fully working anyway.  It would require
substantial reworking for the Registry 3.0 migration because it's closely tied
to the Datastore model and App Engine MapReduce framework, both of which will be
going away.  We can bring back some of these deleted test files as necessary
if/when we rewrite RDE import for the new schema.

On the plus side, in a relational database, RDE import will be much simpler.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231265578
2019-01-28 16:16:36 -05:00
mcilwain
f0c677b18b Rename DNL and SMDRL "login" to "loginAndPassword"
They are passed around in the format username:password, whereas just saying
"login" implies it's just a username and not necessarily also a secret
password. Putting password in the variable name makes it obvious what this is
and reduces the likelihood of anyone ever logging it or otherwise using it
inappropriately.

Note that this does not require data migrations as the actual key used to store
the data in KMS remains unchanged.

This is a follow-up to []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231253964
2019-01-28 16:15:04 -05:00
mcilwain
0130f91830 Add logging to DNL fetch action
This will help us to debug the current MarksDB issue.  This also throws an explicit error earlier when attempting to connect to MarksDB without login credentials being specified, which we know will fail.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231236317
2019-01-28 16:13:27 -05:00
weiminyu
7f08df0d75 Do not complain about empty auto-index file
AppEngineRule throws an exception during teardown if
an empty datastore-indexes-auto.xml is created during
testing. This scenario should be fine, just like if this file
has not been created at all.

This issue causes some flakiness in tests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231229692
2019-01-28 16:11:52 -05:00
gbrodman
5272d8ca7f Make a prettier table to display OT&E check results
We now display the results of each check in addition to the overall result.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231051913
2019-01-28 16:10:16 -05:00
mcilwain
c6e58d3bff Fix some issues caught by IntelliJ static code analysis
The most common issues were:
* Arrays.asList() shouldn't be called with a single parameter.
* Broken Javadoc @links.
* Unnecessary casts and type declarations.
* Unnecessary unused variable initializations.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230994311
2019-01-28 16:08:24 -05:00
gbrodman
3cf26ff9b6 Fix various Error Prone errors that were found by the FOSS build
Most common:
- Unnecessary parentheses and operator precedence clarify (self-explanatory)
- Reference equality--there were a few instances of using == or != improperly
- Qualification of Builder (and similar) imports so that it's clear which type of Builder we're referring to
- Marking some immutable classes with @Immutable since EP desires that all enums be deeply immutable
- String.split() having "surprising behavior"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230971531
2019-01-28 16:05:09 -05:00
shicong
9cd37189c2 Combine dependency update scripts
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230958523
2019-01-28 16:03:29 -05:00
mcilwain
5b0c61dad3 Use simpler Iterables helper when counting domains
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230955066
2019-01-28 16:01:55 -05:00
guyben
5a8760570c Fix RDE reducer broken in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230945703
2019-01-28 16:00:19 -05:00
weiminyu
4213e9246c Update the latest_datastore_export view in Bigquery
This view used to point to the latest datastore backup
using the deprecated export mechanism, which has been
disabled. We will point this view to the new backups.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230930872
2019-01-28 15:58:44 -05:00
mcilwain
e2528875b2 Merge DomainResource into DomainBase
This eliminates the use of Objectify polymorphism for EPP resources entirely
(yay!), which makes the Registry 3.0 database migration easier.

It is unfortunate that the naming parallelism of EppResources is lost between
ContactResource, HostResource, and DomainResource, but the actual type as far as
Datastore was concerned was DomainBase all along, and it would be a much more
substantial data migration to allow us to continue using the class name
DomainResource now that we're no longer using Objectify polymorphism. This
simply isn't worth it.

This also removes the polymorphic Datastore indexes (which will no longer
function as of this change). The non-polymorphic replacement indexes were added
in []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230930546
2019-01-28 15:57:10 -05:00
mcilwain
97c2049669 Catch errors when attempting to delete entities through Objectify
This is currently erroring out on entities that fail to load properly through Objectify (e.g. because their entity type is no longer registered).  The proper thing to do is to catch the error, log it, and fall back to the raw Datastore operation, which will succeed.

The exact Exception this is designed to catch is:

com.google.apphosting.runtime.jetty9.JettyLogger warn: /_dr/admin/deleteEntity (JettyLogger.java:29)
java.lang.IllegalStateException: No registered subclass for discriminator 'DomainApplication'
	at com.googlecode.objectify.v4.impl.PolymorphicEntityMetadata.getConcrete(PolymorphicEntityMetadata.java:133)
	at com.googlecode.objectify.v4.impl.PolymorphicEntityMetadata.load(PolymorphicEntityMetadata.java:164)
	at com.googlecode.objectify.v4.impl.LoadEngine.load(LoadEngine.java:220)
	at com.googlecode.objectify.v4.impl.LoadEngine$1.nowUncached(LoadEngine.java:178)
	at com.googlecode.objectify.v4.impl.LoadEngine$1.nowUncached(LoadEngine.java:164)
	at com.googlecode.objectify.v4.util.ResultCache.now(ResultCache.java:30)
	at com.googlecode.objectify.v4.impl.Round$1.nowUncached(Round.java:73)
	at com.googlecode.objectify.v4.util.ResultCache.now(ResultCache.java:30)
	at com.googlecode.objectify.v4.LoadResult.now(LoadResult.java:25)
	at google.registry.tools.server.DeleteEntityAction.loadOfyEntity(DeleteEntityAction.java:103)
	at google.registry.tools.server.DeleteEntityAction.run(DeleteEntityAction.java:74)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230737553
2019-01-24 20:02:04 -05:00
weiminyu
acbd23fa64 Remove deprecated Datastore backup code
Removed three Action classes and the CheckSnapshot command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230545631
2019-01-24 19:58:54 -05:00
gbrodman
701ebc6a28 Fix a few stylistic issues in preparation for Checkstyle addition
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230524735
2019-01-24 19:57:14 -05:00
jianglai
2c2b1da836 Split py_binary into py_binary and py_library to avoid having py_binary in deps.
Having py_binary in deps is deprecated and will break in 19Q1.
[] for more details.

If this CL broke you, please fix forward by manually running
[]/devtools/python/janitor:lib_split_binary []

Tested:
    TAP --sample for global presubmit queue
    []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230359276
2019-01-24 19:55:35 -05:00
mcilwain
a66ba5310b Include the premium/reserved list tests in external release
This was missed in []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229994119
2019-01-18 17:32:15 -05:00
mcilwain
40540b1846 Add non-polymorphic DomainBase indexes
These indxes do not include the ^i field like the current ones do.

We'll need these so that we can properly index into the DomainBase entities once
DomainResource is merged into them (which will cause us to lose the value of the
^i field).

The migration plan is to push next week's release with these new indexes, then
run the re-save all EPP resources action to populate them, and then eventually
remove the ^i versions of the indexes, retaining only these newly added ones,
once the DomainBase/DomainResource merge is accomplished.

TESTED=Deployed to alpha, ran the re-save all EPP resources mapreduce, and
verified that the new indexes were populated.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229992464
2019-01-18 17:30:43 -05:00
weiminyu
80e71de723 Add debug logging to AppEngineRule
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229983437
2019-01-18 17:29:11 -05:00
gbrodman
5f87c3bff3 Add a button in the admin panel to check OT&E status of a registrar
For now, it only displays a status of "Passed: true|false" or an error message in simple text. In further work we will make the UI nicer.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229971564
2019-01-18 15:35:40 -05:00
mmuller
87ab149049 Externalize premium/reserved list tests
Remove the GoogleInternal designator on the premium/reserved list tests, move
the lists themselves under the config/files directory, and fix all of the
surrounding infrastructure.

The lists do not get exported.  Only the "example.txt" list does, allowing the
test to function on the open source build.

TESTED:
  Verified that the gradle build works and that only the example.txt file is exported.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229940659
2019-01-18 15:34:06 -05:00
weiminyu
d3e25cd5b1 Enable Test Executor sharing again in Gradle build
Identified a few more tests that are impacted by test cleanup
issues outside of Nomulus code base, more likely thread pool
cleanup behaviors in AppEngine testing fixture.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229934266
2019-01-18 15:32:33 -05:00
jianglai
52212748ca Update documentation on nomulus tool authentication
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229780750
2019-01-17 19:22:25 -05:00
mcilwain
d2ee63cf69 Consolidate Dagger modules for utils classes
There was no reason to have several different modules all providing a single
thing. This approach, which creates a single UtilsModule for everything in the
util package, is cleaner. This also removes provisioning of Random and
StringGenerator objects in RegistryConfig.ConfigModule, which don't belong
there because they aren't configuration options.

This also removes insecure random entirely; it was only used in a
single place to generate 24 bytes a couple times per day. We can live with the
lower speed if it means we don't have to worry about multiple types of Random,
or possibly using an insecure random accidentally in a place that security
actually does matter.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229751915
2019-01-17 19:20:52 -05:00
guyben
9aa7b69921 Add web console for creating registrars
This console is only to be used by Admins (either GAE admins for this project, or Support accounts). It is for "internal" use only, not for use by the registrars themselves.

To prevent abuse, the registrar is created in a non-functional PENDING state and can only be made functional from the nomulus shell tool.

While in "PENDING" state, the registrar can be updated from the registrar-console by admins.

Also - moving all the web consoles to the same directory (moving the otesetup/* files into registrar/)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229681011
2019-01-17 19:19:09 -05:00
weiminyu
758dcb5c39 Fix Gradle lint's dependency warnings
This change fixes the following warnings when applied to cloned
github nomulus:

warning   undeclared-dependency              one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly

warning   undeclared-dependency              one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly

warning   unused-dependency                  one or more classes in com.google.http-client:google-http-client:1.28.0 are required by your code directly

warning   unused-dependency                  one or more classes in com.google.api-client:google-api-client:1.28.0 are required by your code directly

✖ 4 problems (0 errors, 4 warnings)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229629444
2019-01-17 19:17:32 -05:00
weiminyu
1e218d1d5b Delete entity synchronously in test method setup
Observed in Gradle test when it seems that the mutating command is
executed before deletion has taken effect.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229585754
2019-01-17 19:15:57 -05:00
guyben
8cfb748316 Remove TLD @Parameter from inside the DnsMetrics
Currently, the TLD is a "hidden" part of the metric - we record it, but by looking at the call site you can't see that we record it.

Also, it's injected from the query / POST parameter, so we might not even be aware of what the value is and it might not be the value we wanted.

Instead, making it explicit in the Metric call. That way it's also more similar to the "logging" statements that record the same data but have to explicitly output the TLD.

It also makes the tests better, as we test that we indeed record the correct TLD

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229550115
2019-01-17 19:12:41 -05:00
weiminyu
bc798f33e9 Automated g4 rollback of changelist 229414759.
*** Reason for rollback ***

Found more tests failing.

*** Original change description ***

Reenable Test Executor sharing in Gradle build

Combining all tests in one suite and drop the forkEvery=1 directive.
Issue was fixed by [] and []

TESTED=Run locally with maxParallelForks =1 and 5, and tested on travis
       with maxParallelForks=5

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229430425
2019-01-17 19:09:29 -05:00