Commit graph

1437 commits

Author SHA1 Message Date
jianglai
23c9cf926c Set namespace as default
This gets around a bug in Spinnaker where the namespace, if missing in the manifest, is set to "spinnaker".

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192825895
2018-04-23 14:39:09 -04:00
jianglai
bee77f0cc3 Make domain check return availability regardless of fee extensions
This allows a registrar to check domain availability even if the fee extension is not set. To obtain the premium price or create the domain, the fee extension is still needed.

PS: I believe the previous tests are erroneous anyway. It is the presence of fee extensions in the check request that we were looking for. Checking if the extension is declared during login has no bearing on the results.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192778137
2018-04-23 14:36:02 -04:00
mcilwain
078e9cbe53 Add better RDE logging for when contact resources don't exist
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192706560
2018-04-23 14:32:59 -04:00
mcilwain
613b19799a Increase commit log bucket count in production to match other envs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192614234
2018-04-23 14:29:59 -04:00
mcilwain
96f66eacd5 Add logging statement to export commit log diffs action
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192548282
2018-04-23 14:28:27 -04:00
mcilwain
168a23206d Increase export-snapshot queue rate from 5/m to 1/s
Five per minute just isn't working well enough on environments with lots of
entities (e.g. alpha and sandbox right now), and there doesn't seem to be a
real need to enforce such a low throttle.  The mapreduce queue, for instance,
has 500/s (effectively no throttle).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192474962
2018-04-23 14:26:55 -04:00
guyben
bec24a3558 Complete enums on tab
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192451820
2018-04-23 14:25:21 -04:00
guyben
8a9453f476 Replace registrar-premium-price-ack with registrar-settings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192355664
2018-04-23 14:22:18 -04:00
guyben
eb17851cb3 Test that update works for every field in RegistrarSettings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192348329
2018-04-23 14:20:43 -04:00
larryruili
ea995cf801 Make lock index parameters non-optional
The optional code has been around for a while, we can get rid of it now.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192344612
2018-04-23 14:19:10 -04:00
guyben
3dfd141e0f Fix the shell to allow multiple mutation of the same entity
Because of the objectify cache, after the first "read, change, write" of the
first mutation - the second mutation would "read" the original value, which
would then fail to be written with the error "Entity changed since init()".

This was specifically seen in the "UpdateRegistrarCommand", but likely affected
other commands as well.

Clearing the cache before each command solves this issue.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192328213
2018-04-10 17:08:45 -04:00
mcilwain
e0c32337fd Add mapreduce to delete load test data
This hard-deletes all contacts and hosts owned by a specific set of registrar
client IDs, currently just "proxy".

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192325211
2018-04-10 17:07:15 -04:00
mcilwain
0923c89981 Fix verb order in nomulus tool domain check commands
This makes them consistent with every other command, which uses the format
verb_noun.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192301468
2018-04-10 17:04:07 -04:00
mcilwain
8f1848e32e Disable verify entity integrity mapreduce on sandbox
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192289233
2018-04-10 16:59:28 -04:00
mcilwain
a8b6195ce2 Make RDE run less frequently on sandbox/alpha
This also removes RDE tasks that shouldn't/can't run on non-production environments, like upload/reporting.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192177779
2018-04-10 16:56:22 -04:00
guyben
38bf86c0fd Incorporate some of the fixes done in RegistrarPremiumPriceAckAction
This is in preparation for merging and then removing
RegistrarPremiumPriceAckAction.

This includes:

test that the data the UI sent isn't stale
---------------------------------------------
Our system is "read, modify, write". However, if between the "read" and the "write" someone else changed the registry, my write will undo their change even if I didn't touch any of their fields.
To solve that - we use the "lastUpdateTime" timestamp of the registrar. the UI reads it with the rest of the data, and sends it back on "write". We will now make sure the registrar currently in datastore has the same timestamp.

support premium-price-ack flag
---------------------------------
Add support for reading and writing this flag. We still won't be using it - that's in a followup CL, but we support it.

support changing the URL
------------------------
Add changing the URL in the UI, under the "whois" section

Will replace the Ack endpoint with this (and remove that endpoint) in a followup CL

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192154078
2018-04-10 16:54:51 -04:00
mcilwain
3bbaf585e5 Don't prompt to confirm non-mutating nomulus EPP tool commands
This is accomplished by making all non-mutating commands function with dry run set
to true, which also has the pleasurable side effect of not prompting for dry-run
mutating commands either, which also do nothing different/special on the second
run.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192149150
2018-04-10 16:53:18 -04:00
guyben
013558c814 Make it VERY clear when nomulus shell is on PROD
We don't want people to accidentally run commands on prod thinking they were on
Alpha / Sandbox.

To do that - we add 2 safeguards:

1) when on prod, the shell has a strong RED "PRODUCTION" in the commandline, while on alpha/sandbox it's green.

2) if a prod shell is idle for > 1h, it exits. So don't accidentally use a prod shell from a long time ago.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191931731
2018-04-10 16:51:42 -04:00
guyben
7bf0b059a6 Make the example whitelist IP be legal
Currently the example whitelist IP is 1.1.1.1/24, which is illegal. Changed to
1.1.1.0/24, which is legal

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191900036
2018-04-10 16:50:10 -04:00
guyben
6d5f7dc4a1 Print documentation for flags
After writing a flag on the shell, pressing "tab" will print out the
documentation for that flag.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191899137
2018-04-10 16:48:33 -04:00
jianglai
983bd27ee0 Read GCP proxy EPP SSL secret from GCS
This allows us to not ship the proxy with certificates/private keys. The secret is still encrypted by KMS. Reading the secret only happens once when the first EPP request comes in, which should not incur any tangible performance penalty.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191771680
2018-04-10 16:38:31 -04:00
jianglai
18a145eef1 Use self signed certificate when running the proxy locally
This allows us to not obtain a certificate and encrypt it with KMS when running the proxy locally during development.

Also updated FOSS build dagger version.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191746309
2018-04-10 16:36:56 -04:00
guyben
ea891001d9 Fix registrar security console
The registrar security console failed because it assumed the email is a
required field for the registrar, but it isn't (at least - create_registrar
doesn't require an email, and update_registrar lets you remove the email).

Fixed by allowing it to *remain* unset if it was unset originally, but if it was set - it's required.

There are more fixes needed, but they aren't related to the email, so they will wait for the next CL

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191623034
2018-04-10 16:35:21 -04:00
mcilwain
e816913c61 Increase # of commit log buckets ~4X for all non-prod environments
This also reduces the interval of the commitLogCheckpoint cron job to once
every three minutes, as this job needs to load all commit log bucket entities.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191613858
2018-04-10 16:33:47 -04:00
guyben
3338b91c84 Remove the "shell" command from the completions, and allow empty lines
Even though you couldn't run a "shell" inside a "shell", the completion still
assumed you could :(

On the way - fixing error on empty lines: when you just press "enter", the shell should ignore it rather than try to run it as a command (and getting an error, obviously)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191605029
2018-04-10 16:32:12 -04:00
jianglai
4c06b36118 Format terraform files
For some reason the auto-formatting didn't happen when these files are first checked in.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191589487
2018-04-10 16:27:23 -04:00
jianglai
e7f033201b Use process substitution in terraform config script
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191584425
2018-04-10 16:25:36 -04:00
mcilwain
07d38340f3 Cache domains, contacts, and hosts in WHOIS queries
This should prevent having issues with hot key paths on entities that
experience a heavy WHOIS volume (e.g. contacts that registrars reuse on
many domains).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191506124
2018-04-10 16:24:03 -04:00
guyben
cfd83ad4dc Add "libreadlines"-like behavior to the shell command
Using the jline open-source library.

We save the history between invocations to ~/.nomulus_history

We add some simple completions:
- first argument completes to command name
- all other arguments complete to the command parameters, or filename

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191501023
2018-04-10 16:22:30 -04:00
mmuller
3684fa3366 Create prod Cloud DNS domains in prod zonespace
When running create_cdns_tld in "production" mode, specify the Cloud DNS
production namespace instead of the staging namespace.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191451390
2018-04-10 16:20:52 -04:00
jianglai
839e4aa682 Remove some fields from WHOIS output
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191445626
2018-04-10 16:17:42 -04:00
mcilwain
377fe5f573 Allow number of commit log buckets to be increased
Also increases the number of commit log buckets on alpha to 397 and correspondingly
reduces the frequency of commit log diff exporting to once every 3 minutes.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191440586
2018-04-10 16:16:08 -04:00
guyben
951ba78cef Add logging on input failure for registrar-settings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191369657
2018-04-10 16:14:29 -04:00
guyben
70efdc1cb7 Add counters to the RdeStaging mapreduce
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191339016
2018-04-10 16:09:46 -04:00
mmuller
3dff2ba4c7 Inline calls to get command from jcommander
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191338771
2018-04-10 16:08:10 -04:00
mmuller
18290911a0 Fix multiple invocations of the "shell" command
JCommander doesn't seem to reset objects when it populates them with data from
an argument list during command processing, so recreate the command objects
every time we do a run().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191332392
2018-04-02 16:55:15 -04:00
mcilwain
54a8cd09ea Use some Java 8 features in commit log code
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190985373
2018-04-02 16:53:46 -04:00
jianglai
c40eda3235 Rename WhoisServer and WhoisHttpServer to actions
These servlets are converted to actions during daggerization. Calling them servers are misleading.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190942237
2018-04-02 16:52:14 -04:00
mcilwain
038a2679c7 Add per-test output to verify_ote command in --summarize mode
Before this change the output looks like this:

registrar1      - Num actions:   93 - Reqs passed: 16/16 - Overall: PASS
registrar2      - Num actions:   47 - Reqs passed:  6/16 - Overall: FAIL

After this change the output looks like this:

registrar1      - # actions:   93 - Reqs: [----------------] 16/16 - Overall: PASS
registrar2      - # actions:   47 - Reqs: [...--.-...-...--]  6/16 - Overall: FAIL

The status of each test is displayed as a hyphen (passing) or a period (failing),
and the tests are always displayed in the same order so it's easier to get an overall
view of whether registrars are struggling with the same tests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190776935
2018-04-02 16:50:42 -04:00
mmuller
e1ad4d663c Remove Registrar caching from all console actions
Caching turns out to be an anti-pattern for the console.  If we use it, changes from the user just get obliterated by the older, cached version the next time the console refreshes (and it happens to refresh after every update).  Caching is also not very useful here, as the amount of database access driven by the console is very small.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190650931
2018-04-02 16:49:00 -04:00
jianglai
6dec95b980 Use terraform to config GCP proxy setup
With terraform (https://terraform.io) we can convert most of the infrastructure setup into code. This simplifies setting up a new proxy as well as providing reproducibility in the setup, eliminating human errors as much as possible.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190634711
2018-04-02 16:46:01 -04:00
guyben
2bbde9d9a9 Retry any Datastore reads in EppResource map-reduce input
Datastore has a non-zero chance of failing on reads. A map-reduce with too many
failures will eventually give up. As a result, any map-reduce that goes over a
large number of datastore entities is almost guaranteed to fail.

Since we expect to have a large number of EppResources, we make sure to wrap
all datastore reads with some retrying mechanism to reduce the number of
transient failures that propagate to Map-Reduce.

This feature already existed for CommitLogManifestReader, we refactor the code to use the same retrying mechanism in EppResource readers.

Also removed the transactNew around the reads because looking at the source - it doesn't actually do anything we need (doesn't retry on any failure other than concurrency failure)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190633281
2018-04-02 16:44:29 -04:00
guyben
c1ca88ac9e Add OT&E verification that a domain was successfully created in sunrise
We already have verifications that a domain application was created in sunrise
- which checks for end-date sunrise. Start-date sunrise has checks that a
domain (not application) was created. There's no need to specifically check for
a signed mark, since a successful domain create during sunrise must have a
signed mark in it.

Also removed the requirement for end-date sunrise / landrush testing.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190551080
2018-04-02 16:41:28 -04:00
mcilwain
fea495e319 Log registrar IP address whitelist when auth is denied
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190462466
2018-04-02 16:38:28 -04:00
mcilwain
0154cf09bb Change dash to underscore in param name for consistency
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190261145
2018-04-02 16:36:50 -04:00
guyben
89d8ba93f2 Remove transition code from []
The parameters were optional during the transition to allow old jobs stuck in the queue to work properly. It's been 2 months now so it's time to end the transition.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190235532
2018-04-02 16:35:20 -04:00
mmuller
785225fc28 Implement "premium price ack required" checkbox
Implement a checkbox in the "Resources" tab to allow registrars to toggle
their "premium price ack required" flag.

Tested:
  Verfied the console functionality by hand.  I've started work on an
  automated test, but we can't actually test those from blaze and the
  kokoro tests are way too time-consuming to be practical for development, so
  we're going to have to either find a way to run those locally outside of
  the normal process or make do without a test.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190212177
2018-04-02 16:33:51 -04:00
mcilwain
f47965d5f6 Consolidate reserved terms export disclaimer
We're now publishing almost everything, rather than holding back the
ICANN reserved terms (there's no point in doing so as those aren't secret).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190149405
2018-04-02 16:32:21 -04:00
guyben
b5ae37c5cc Return more informative errors when signed mark is invalid at this time
A "mark" tells us that the holder owns the trademark for a given domain name. It is signed for authentication.

If the signature's certificate is either "not yet valid" or "expired", we return explicit errors to that effect.

But in addition to the signature's certificate, the mark itself might not be valid yet or already expired. Right now if that happens - we return an error saying "the mark doesn't match the domain name".

That is wrong - as the mark can match the domain name, just be expired. Returning "the mark doesn't match the domain name" in that case is misleading.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190069976
2018-04-02 16:29:24 -04:00
jakubvrana
27dedf316b Mark <iframe src=""> with |blessStringAsTrustedResourceUrlForLegacy.
Soy is about to require trusted_resource_url in <iframe src="">. This change prepares for that by blessing existing string URLs.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190054283
2018-04-02 16:27:52 -04:00