Commit graph

248 commits

Author SHA1 Message Date
mcilwain
a111ff0b0a Rename "absent" to "empty" in test names
This is left over from the transition from Guava to Java 8 Optionals.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207154260
2018-08-10 13:46:48 -04:00
mcilwain
f614044681 Add an auto update time field to the Cursor entity
Also adjusts the nomulus list_cursors command to output the value of this field.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206646117
2018-08-10 13:44:25 -04:00
mcilwain
6e74ba0587 Wrap ModulesService in new AppEngineServiceUtils
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).

This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.

This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
2018-08-10 13:44:25 -04:00
mcilwain
a2fe058865 Allow square bracket expansion when specifying nameservers
I'm finally fed up enough with all the nameserver changes we've had to make on our
self-allocated domains to improve the command.  Now you can simply run:

$ nomulus ... update_domain ... -n ns[1-4].foo.bar

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205282317
2018-08-10 13:44:25 -04:00
guyben
8ec2eaf39c Simplify the RyDE API
Second step of RDE encoding refactoring.

Creates a single OutputStream encode RyDE files.
This replaces the 5 OutputStreams that were needed before.

Also removes all the factories that were injected. It's an encoding, there's no point in injecting it.

Finally, removed the buffer-size configuration and replaced with a static final
const value in each individual OutputStream.

This doesn't yet include a decoder (InputStream). And there's still a lot of overlap between the Ryde and the Ghostryde code. Both of those are left for the next CLs.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204898369
2018-07-17 22:03:53 -04:00
mcilwain
8942c4fad1 Fix add/remove calculations when updating multiple domains
Lists used as accumulators were being updated individually for each domain
without starting over from a fresh list each time, so the number of changes
would grow for each additional domain and potentially be wrong if the previous
domains were set up differently.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204526006
2018-07-14 01:37:03 -04:00
mcilwain
278ec2b289 Pretty-print actual XML in tests
This means that, when writing new tests that are failing, you get much more
useful logs that show the actual XML in a more comprehensible format that is
suitable for pasting back into the golden file in the test (if the change was
intended).

This requires outputting the standalone parameter in the XML transformer, and
some minor changes to some tests as a result that were relying on it being
stripped out.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204513690
2018-07-14 01:37:03 -04:00
mcilwain
df5d9957f7 Move the AllocationToken entity to the token subpackage
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204485366
2018-07-14 01:37:03 -04:00
mcilwain
32b3563126 Delete all Braintree code
We never launched this, don't planning on launching it now anyway, and it's rotted over the past two years anyway.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202993577
2018-07-14 01:37:03 -04:00
mcilwain
a4e99b18e2 Clean up internal imports in BUILD files
This removes the following unnecessary imports:

//third_party/java_src/gtld/java/google/registry/bigquery
//third_party/java_src/gtld/java/google/registry/config
//third_party/java_src/gtld/java/google/registry/cron
//third_party/java_src/gtld/java/google/registry/dns
//third_party/java_src/gtld/java/google/registry/gcs
//third_party/java_src/gtld/java/google/registry/mapreduce
//third_party/java_src/gtld/java/google/registry/model
//third_party/java_src/gtld/java/google/registry/module/backend
//third_party/java_src/gtld/java/google/registry/module/frontend
//third_party/java_src/gtld/java/google/registry/module/pubapi
//third_party/java_src/gtld/java/google/registry/module/tools
//third_party/java_src/gtld/java/google/registry/request
//third_party/java_src/gtld/java/google/registry/security
//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers
//third_party/java_src/gtld/java/google/registry/util
//third_party/java_src/gtld/java/google/registry/xjc
//third_party/java_src/gtld/javatests/google/registry/model
//third_party/java_src/gtld/javatests/google/registry/testing
//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce

The exact command run to generate this CL was:

build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java_src/gtld/java/google/registry/bigquery,//third_party/java_src/gtld/java/google/registry/config,//third_party/java_src/gtld/java/google/registry/cron,//third_party/java_src/gtld/java/google/registry/dns,//third_party/java_src/gtld/java/google/registry/gcs,//third_party/java_src/gtld/java/google/registry/mapreduce,//third_party/java_src/gtld/java/google/registry/model,//third_party/java_src/gtld/java/google/registry/module/backend,//third_party/java_src/gtld/java/google/registry/module/frontend,//third_party/java_src/gtld/java/google/registry/module/pubapi,//third_party/java_src/gtld/java/google/registry/module/tools,//third_party/java_src/gtld/java/google/registry/request,//third_party/java_src/gtld/java/google/registry/security,//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers,//third_party/java_src/gtld/java/google/registry/util,//third_party/java_src/gtld/java/google/registry/xjc,//third_party/java_src/gtld/javatests/google/registry/model,//third_party/java_src/gtld/javatests/google/registry/testing,//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202652421
2018-07-14 01:37:03 -04:00
mcilwain
bb1b4bc0d4 Clean up third_party imports in BUILD files
This removes the following unnecessary imports:

//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8

The exact command run to generate this CL was:

build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
2018-07-14 01:11:51 -04:00
mcilwain
43ed2cd7b3 Clean up annotation imports in BUILD files
This affects JSR305, JSR330, and Guava annotations.

The exact command run to generate this CL was:

build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
2018-06-27 15:28:53 -04:00
guyben
6ff48b7dae Simplify the Ghostryde API
First step of RDE encoding refactoring.

Creates a single InputStream (OutputStream) to decode (encode) Ghostryde files.
This replaces the 3 InputStreams (OutputStreams) that were needed before.

Also removes a lot of classes, and removes the "injection" of the Ghostryde
class. It's an encoding, there's no point in injecting it.

Finally, removed the buffer-size configuration and replaced with a static final
const value. It's just a buffer size - it doesn't actually affect much. There
are much more "important" fields that weren't configured (such as the
compression algorithm and whether or not to do integrity checks)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202319102
2018-06-27 15:28:53 -04:00
mcilwain
892c1fc707 Update signed marks files and add an expiration test
We'll continue to use injected clocks for the rest of our tests that use signed marks files, so that they don't all fail after the current validity period. The new test TmchTestDataExpirationTest will let us know when the files are expired, so we can update them.

All updated test data files come from https://newgtlds.icann.org/en/about/trademark-clearinghouse/registries-registrars

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202208196
2018-06-27 15:28:53 -04:00
guyben
b4cb3e0a6f Inject clock for Tmch validity using InjectRule instead of manually
Basically a less patchy version of []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202138086
2018-06-27 15:28:53 -04:00
guyben
44c517f111 Set clock used to validate Tmch signature in tests
The signature has expired on Mon Jun 25 23:59:59 UTC 2018.

Instead of fixing all the test files with a new signed mark, we inject the
clock used to validate the signature.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202116721
2018-06-27 15:28:52 -04:00
mcilwain
4c7bc3b18c Improve internal build system speed
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202016862
2018-06-27 15:28:52 -04:00
mcilwain
6706b99828 Delete remnants of registrar-level credits
We never fully used this stuff but definitely no longer use it following our
recent billing refactor. It's confusing to retain all of these entities and
commands given that none of them are actually used by anything.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201978094
2018-06-27 15:28:52 -04:00
mcilwain
a5cc359813 Make nomulus list_cursors command faster by batching loads
They're all in the same entity group anyway (the cross-TLD one), so they can be
loaded in a single call instead of individually.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201364854
2018-06-27 15:28:52 -04:00
mcilwain
ad73f3d167 Remove more unnecessary "throws" declarations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201243722
2018-06-27 15:28:52 -04:00
mcilwain
8b263baefa Delete MultiplyingCloudDnsWriter
Now that the large zone re-signing test is complete, we no longer need it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
2018-06-27 15:28:06 -04:00
mcilwain
5d80f124ca Remove unnecessary "throws" declarations
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201058582
2018-06-18 18:17:56 -04:00
mcilwain
4098487b80 Fix passing of null to some parameters expecting Optional
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200465575
2018-06-18 18:09:21 -04:00
guyben
646c6fa069 Print GAE-UserId even when null
Also explicitly state that contacts missing GAE-UserId can't access the
registrar console

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200402953
2018-06-18 18:06:23 -04:00
mcilwain
ddf55005c3 Allow nomulus list_domains to query any number of TLDs
This limit did not exist prior to [] which added the ability to limit
the size of the list. I didn't think that we needed to be able to query more
than 30 TLDs at any one time so I got rid of batching, but it turns out we do
need this ability for domain_watcher. So I'm re-adding batching, which is a
little bit more complicated now that we're also limiting and sorting by creation
time.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199826414
2018-06-18 17:48:44 -04:00
mcilwain
5c7a20797e Default clientId to registry's registrar in domain lock commands
All domain locks we've processed so far are as a result of the URS process, for
which the clientId is always that of the registry's registrar. So it makes sense
to default to that value, while still retaining the option to specify it if
required in case we ever support registrar-requested registry locks in the
future.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199350120
2018-06-06 15:17:36 -04:00
mcilwain
897690119e Add a nomulus renew_domain command
This may come in handy when dealing with URS requests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199332085
2018-06-06 15:16:09 -04:00
mcilwain
7c0b8cab0b Add premium support to nomulus create_domain command
Premium prices are automatically detected and set, with an informational
message displayed to the user prior to executing the command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199223541
2018-06-06 15:11:44 -04:00
guyben
9d2b1e7572 Consolidate all Set parameter parsing
Currently, we have two different ways to parse a "set" parameter:
key=value1&key=value2&key=value3...
and
keys=value1,value2,value3

This is error prone for several reasons:
- different parts of the code must be "synchronized" to use the same style (the
  place that creates the request, and the place that parses the request)
- for the key=value1&key=value2, we often use the same key name for the single
  value and the set value. This can result in subtle bugs where part of the
  code will successfully read the key assuming there's only one key (and will
  get the first key=value1, ignoring the rest)

Here we transition everything to the keys=value1,value2,value3 method. This one
was chosen because:
- it's shorter
- it's more intuitive for users
- the key name is plural, differentiating it from the singular key=value that
  other requests might need

-----------------------------------

To make sure there are not "transition issues", we will continue to support
(with warnings) the key=value1&key=value2 parameter parsing until we're sure we
haven't forgotten to update any part of the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198810681
2018-06-06 15:04:02 -04:00
mcilwain
92190f8699 Merge the check_domain_fee nomulus command into check_domain
I'm tired of running check_domain only to then realize I needed to run
check_domain_fee instead because the domain ended up being premium. We require
the use of the fee extension to register domains on all of our TLDs anyway so we
might as well always be sending the fee extension when checking domains
too. There's no additional harm in sending along the fee extension (it only
sends back additional information that is often useful), so just make that the
default and remove the extra unnecessary command.

Note that check_domain_claims can't be merged in too because it fundamentally
works differently. It doesn't query the availability of domain names for
registration, just whether they're trademarked.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097326
2018-05-30 12:18:54 -04:00
mcilwain
674a914afc Add flag to delete domain immediately in nomulus too
We've already had the need to do this on a few occasions, so it's best to make
it easy rather than requiring hand-generated XML all the time.

Also normalizes the boolean --registrar_request parameter to not have arity=1.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198093829
2018-05-30 12:18:54 -04:00
mcilwain
8f456bcf64 Clarify when to use cache (or not) when loading premium lists
You don't want to use the cache when loading them for the purposes of updating
them, but you definitely do still want to use the cache when checking the
price of individual domains.

In [] the cache clearing of premium lists on update was removed. This
is a good thing in aggregate because the cache is per-instance and thus
misleading, but it also caused us to not be able to update the same premium
list twice within an hour because the second update would hit a "PremiumList
was concurrently edited" exception, owing to first loading the stale version
from the cache for the purposes of updating it. Now we bypass the cache for
that purpose.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197768142
2018-05-30 12:18:54 -04:00
mcilwain
9c0d3b6db3 Add limit to list_domains command
This allows list_domains to continue working for large TLDs.

TESTED=Deploys to alpha and it works to list the most recently created domains even
on a TLD with a huge number of domains on it (much more than .app has currently).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196717389
2018-05-17 21:52:35 -04:00
mmuller
25d03f239c Allow "encapsulated output" from the shell command
Since the ConsoleReader now controls the display of the prompt, there is no
longer any way for an external program interacting with the nomulus shell to
recognize when the last command issued has been completed.

To remedy this, we introduce an "--encapsulate_output" flag, which causes
standard output and standard error to be wrapped in a class that precedes
all normal lines of output with a prefix ("out: " or "err: ", accordingly)
and allows the command processor to insert a "SUCCESS" or "FAILURE"
line (with no special prefix) after completing the processing of a
command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196702338
2018-05-17 21:52:35 -04:00
mcilwain
c242a4d08f Add nomulus count_domains command
This efficiently counts domains without having to load them (as opposed to the
existing list_domains command which does load then).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194559095
2018-05-05 23:37:11 -04:00
guyben
bec24a3558 Complete enums on tab
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192451820
2018-04-23 14:25:21 -04:00
mcilwain
0923c89981 Fix verb order in nomulus tool domain check commands
This makes them consistent with every other command, which uses the format
verb_noun.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192301468
2018-04-10 17:04:07 -04:00
mcilwain
183dae6e80 Migrate away fully from MockitoJUnitRunner
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192291786
2018-04-10 17:01:04 -04:00
mcilwain
3bbaf585e5 Don't prompt to confirm non-mutating nomulus EPP tool commands
This is accomplished by making all non-mutating commands function with dry run set
to true, which also has the pleasurable side effect of not prompting for dry-run
mutating commands either, which also do nothing different/special on the second
run.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192149150
2018-04-10 16:53:18 -04:00
guyben
013558c814 Make it VERY clear when nomulus shell is on PROD
We don't want people to accidentally run commands on prod thinking they were on
Alpha / Sandbox.

To do that - we add 2 safeguards:

1) when on prod, the shell has a strong RED "PRODUCTION" in the commandline, while on alpha/sandbox it's green.

2) if a prod shell is idle for > 1h, it exits. So don't accidentally use a prod shell from a long time ago.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191931731
2018-04-10 16:51:42 -04:00
guyben
6d5f7dc4a1 Print documentation for flags
After writing a flag on the shell, pressing "tab" will print out the
documentation for that flag.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191899137
2018-04-10 16:48:33 -04:00
guyben
6699915132 Move tests to use TestDataHelper for reading resources
TestDataHelper is build exactly to prevent direct reads of resources. It caches
the resources and makes sure they are in the correct directory.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191785004
2018-04-10 16:43:17 -04:00
guyben
3218a9b77e Replace domain allocation in the OTE test with "create with dsdata"
We no longer want to check end-date sunrise for OTE verification, so we should
remove any reference to allocation (and applications, which was removed in a
previous CL)

However, this last allocate was also the command that tested secDNS. So we need
to replace it with a regular create with secDNS.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191604222
2018-04-10 16:30:39 -04:00
guyben
cfd83ad4dc Add "libreadlines"-like behavior to the shell command
Using the jline open-source library.

We save the history between invocations to ~/.nomulus_history

We add some simple completions:
- first argument completes to command name
- all other arguments complete to the command parameters, or filename

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191501023
2018-04-10 16:22:30 -04:00
mmuller
87a58dbcae Remove unused variable from ShellCommandTest
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191340225
2018-04-10 16:11:22 -04:00
mmuller
18290911a0 Fix multiple invocations of the "shell" command
JCommander doesn't seem to reset objects when it populates them with data from
an argument list during command processing, so recreate the command objects
every time we do a run().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191332392
2018-04-02 16:55:15 -04:00
mcilwain
038a2679c7 Add per-test output to verify_ote command in --summarize mode
Before this change the output looks like this:

registrar1      - Num actions:   93 - Reqs passed: 16/16 - Overall: PASS
registrar2      - Num actions:   47 - Reqs passed:  6/16 - Overall: FAIL

After this change the output looks like this:

registrar1      - # actions:   93 - Reqs: [----------------] 16/16 - Overall: PASS
registrar2      - # actions:   47 - Reqs: [...--.-...-...--]  6/16 - Overall: FAIL

The status of each test is displayed as a hyphen (passing) or a period (failing),
and the tests are always displayed in the same order so it's easier to get an overall
view of whether registrars are struggling with the same tests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190776935
2018-04-02 16:50:42 -04:00
guyben
c1ca88ac9e Add OT&E verification that a domain was successfully created in sunrise
We already have verifications that a domain application was created in sunrise
- which checks for end-date sunrise. Start-date sunrise has checks that a
domain (not application) was created. There's no need to specifically check for
a signed mark, since a successful domain create during sunrise must have a
signed mark in it.

Also removed the requirement for end-date sunrise / landrush testing.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190551080
2018-04-02 16:41:28 -04:00
mcilwain
0154cf09bb Change dash to underscore in param name for consistency
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190261145
2018-04-02 16:36:50 -04:00
mmuller
f1c29633fb Add a "shell" pseudo-command to nomulus tool
Add the "shell" command which lets you run multiple other command in a single
session, sparing you the initialization costs for all but the first of them.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=188712815
2018-03-19 18:22:01 -04:00