Moving to KmsKeyring, any command that uses the Keyring must be a
RemoteApiCommand to work.
TESTED: successfully encrypted a file with no logged errors.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155673003
My continuing investigations into necessary resources for running
our environments seems to indicate that four instances should be
sufficient for our purposes. If it's not, we can always revert.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155607688
This should be %SOURCE_DATASET% so that it's adjusted appropriately based on the parameter value.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155582718
The absence of these fields causes RDE failures, so they are in effect
required on any functioning registry system. We are currently
experiencing problems in sandbox caused by null values on these fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155474895
These annotations are part of guava's API, and they cannot safely be
optimized off the compile-time classpath.
This works around google/error-prone#615.
See []
H/T @cushon
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155433554
Frontend validation: ensures that only one WHOIS abuse contact exist per registrar. Any existing WHOIS abuse contact will be overridden when a new one is designated.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155289097
We've determined that getting correctness semantics right, even
in the few cases that it is possible to do so (see linked bug for
audit) is not worth the bother in terms of highly complicated code
and potential bugs. This CL turns off memcache at the Ofy level
but doesn't rip out the annotations etc. so that we can quickly
turn it back on if this turns out to have been a mistake.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155227761
If there are gaps or forks in the commit log history files, raise an exception
and display the possible sets of files in the log files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155219410
Also added corresponding getters and setters for the new field. Note that
nothing has changed on the RDAP front for now, as the CL&D only concerns WHOIS.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155116134
This change means that RdeStagingAction now accepts POST requests and supports the API auth method, which is needed for the new GenerateEscrowDepositCommand (as of [] to be able to invoke the action in manual mode. Note that this shouldn't affect the cron setting that invokes it currently, which are GET requests with INTERNAL auth.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154764870
This is needed to delete an invalid host resulting from a previous bug.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154757425
The logging for exceptions in FlowRunner - always at WARNING - has long been sub-optimal. For EppExceptions it's too aggressive/spammy to log at WARNING because it's generally not actionable - EppException gets properly thrown for all kinds of ordinary reasons (trying to create a resource when one already exists with that foreign key) and/or for client misbehavior that we can't control (sending bad parameter values, etc.). For non-EppException RuntimeExceptions, it's redundant with existing logging in EppController.
This CL resolves this by removing that logging in FlowRunner entirely in favor of the EppController logging, where we're now logging EppExceptions at INFO in parallel with the existing logging of RuntimeExceptions at SEVERE. This has the benefit that we're now logging EppExceptions that come from FlowPicker (by way of EppExceptionInProviderException), which previously were unlogged.
Note however that this does mean that in places where we run FlowRunner without EppController - exclusively test code as it stands today - we'd no longer be logging EppExceptions. If that seems like a loss, we could either reinstate logging there (at INFO) and just deal with redundant messages for most EppExceptions, or we could add it manually to places where we call FlowRunner.run() in tests and avoid the redundancy that way.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154733365
Currently, if the ID is invalid, parentKey is set to null, causing it to return all history entries. Note that there is still a problem that you cannot look up history entries for entities which have been soft deleted, because the foreign key lookup won't work. That is unfortunate, but at least this simple fix makes things better than they were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154564262
We want to lower the maximum number of service instances as much as
possible without affecting service reliability so that we can make
stronger statements about what the maximum cost of running a typical
Nomulus environment might be. This first step likely won't affect the
frontend and tools modules in practice because they aren't typically
running even this many instances, but it will clamp down on the
number of backend instances (which should be fine; it just means the
mapreduces will take longer).
Alpha is tuned down the same as sandbox for consistency reasons.
This also standardizes on the B4 size (which has 512 MiB RAM) for
all instances. Most instances were already using this, and the
deviations from it were seemingly at random. Crucially, backend,
which is likely most sensitive to this because it uses the mapreduce
library, is already on the smaller memory size.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154537995
This cleans up a TODO introduced in the original bug, which allowed tasks to
have optional Trids in the case it was an older enqueued task.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154477042
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
fail. We aren't ready to do this in prod yet, which is why the tests have a TODO
in them. The new tests fail if you change line 134 in Ofy to not use memcache
and either use the unchanged original flow code, or use the new
inlined code and change loadWithMemcache() to load(). They pass with the new
inlined code that calls loadWithMemcache(), as long as the @Cache is added to
DomainResource.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154457655
Only Ofy itself and its two helpers (AugmentedSaver and
AugmentedDeleter) need to use the real ofy(). All other
callers should be using Ofy. Fixing this even though it
doesn't change anything because I found it baffling to
follow the code while trying to make a small change.
Update: added a presubmit to enforce this.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154456603
- Document how to bring up an environment without starting cron.
- Fix the javascript to be copied into the console to ensure that the user
edits the parameters.
- Link the production build step back to the procedures document.
- Fix a few textual issues.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154419128
Mostly these are calls to ForeignKeyIndex.create() (a static method) via subclasses, which is pretty misleading in this case since the type of the return value has nothing to do with the subclass you're qualifying the static method call with (the returned type depends only on the type of the EppResource parameter).
Note however though that while the style guide indeed prohibits qualifying static member references with things other than the class name, the subclassing case is apparently not considered subject to that prohibition in general:
https://groups.google.com[]d/msg/java-style/8ViX-Rh2_sc/48n2lz5nAAAJ
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154331605
We ran into a bunch of prober deployment issues this past week when
attempting to spin up a new cluster because the newly created prober
TLDs had null values for the dnsWriter field. Given that VoidDnsWriter
exists, we can require that dnsWriter always be set, and have people
use that if DNS publishing is not required.
Also cleans up a bunch of related inconsistent exception messages and
tests not verifying said exception messages properly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154325830
TESTED=The test fails if you change line 134 in Ofy to not use memcache
and use the unchanged original Registry.get() code. This is the
expected behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154226534
TESTED=For all tests, I added @Cache to DomainBase because otherwise the tests will
fail. We aren't ready to do this in prod yet, which is why the tests are still
marked @Ignore. The new tests fail if you change line 134 in Ofy to not use memcache
and either use the unchanged original DomainCreateFlow code, or use the new
inlined code and change loadWithMemcache() to load(). They pass with the new
inlined code that calls loadWithMemcache(), as long as the @Cache is added to
DomainResource.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154224748
This avoids loading all contacts and hosts before the failfast runs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154179321
The requirement will instead be enforced on registrar contact that is visible in WHOIS as abuse contact.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154102111
As part of b/36599833, this makes FlowReporter log the tld(s) of every domain
flow it executes, so we can provide ICANN reporting totals on a per-TLD basis.
It also adds several other fields that we're computing anyway and which seem
useful, particularly for debugging any issues we see in production with the data
that we're attempting to record for ICANN reporting. The full set of fields is:
- commandType (e.g. "create", "info", "transfer")
- resourceType* (e.g. "domain", "contact", "host")
- flowClassName (e.g. "ContactCreateFlow", "DomainRestoreRequestFlow")
- targetId* (e.g. "ns1.foo.com", "bar.org", "contact-1234")
- targetIds* - plural of the above, for multi-resource checks
- tld** (e.g. "com", "co.uk") - extracted from targetId, lowercased
- tlds** - plural of the above, deduplicated, for multi-resource checks
* = only non-empty for resource flows (not e.g. login, logout, poll)
** = only non-empty for domain flows
Note that TLD extraction is deliberately very lenient to avoid the complexity
overhead of double-validation of the domain names in the common case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154070794
This wasn't being recorded correctly because the clientId is only
set in LoginFlow after the flow succeeds, whereas we were
previously logging the clientId before executing the flow. This
adds special handling for LoginFlow. Note that we only set the
metric label to the clientId for valid registrar logins, to ensure
that metric cardinality doesn't grow unbounded (as it might if we
used every arbitrary string passed in as an attempted login).
This also refactors creation and handling of FakesAndMocksModule so
as to be able to make test assertions about EPP metrics from
integration flow tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154048280
Since this feeds into ICANN reporting, we don't want to muddy the data
there with dry-runs, which are always internal-only artifacts of tool usage
and shouldn't really count as real attempts to do SRS actions, since they
are always going to abort with no effect.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153856915
Add build rules for creating "nocron" war and ear files for use during
database restores in production and alpha.
After a little consideration, I think this is the right way to do this. If we
want to set it up in other environments, we'll need to add these rules for
them, too, and at that point it may be worthwhile to wrap the rules we want in
a macro, but for now this is fairly clear and not execessively redundant.
This change works by modifying the zip_file rule to allow us to exclude
"cron.xml". I have verified that:
- The alpha "nocron" files are generated without cron.xml.
- The production "nocron" files are generated without cron.xml.
- I can do a full push to alpha based on deploy_nocron which doesn't install a
new cron.
- The normal production and alpha war-files _do_ contain cron.xml.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153853205
This primarily adds accessors to EppInput that will be used for flow reporting
logging in FlowReporter. Specifically, it adds:
- Optional<String> getResourceType() -> domain/host/contact
- Optional<String> getSingleTargetId() -> for SingleResourceCommands
And in addition, it adjusts getCommandName() so that it's now named
getCommandType() for better parallelism with the new getResourceType() (since
getResourceName() would be misleading), and it changes the value returned to be
lowercased, again for consistency. This isn't an issue because getCommandName()
isn't actually used anywhere right now (it was formerly used for EPP whitebox
metrics, but no longer due to recent changes there).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153851957
This prevents a possible failure mode of the logging where the logged
EPP input XML is very large (which can happen e.g. for domain creates
with large SMD values). In those cases, the XML might cause the overall
JSON string to be too large to fit within a single log entry [1], in which
case it gets split over multiple lines and breaks automatic parsing.
This mitigates that case by logging the EPP input (raw and base64-encoded)
in a separate log statement so that the more compact metadata (like clientId)
and derived values (like ICANN reporting field) will still be in an intact
JSON string even in that case, and can still be readily parsed. It's okay
if the actual EPP XML is harder to parse, since once we're logging the right
metadata fields we shouldn't need to automatically parse the EPP XML in any
normal cases.
[1] I haven't found this exact limit or splitting algorithm, or whether it's
a property of java logging or GAE log ingestion. The GAE logs page does note
that a single application log entry (within a request, which can have up to
1000 such entries) maxes out at 8KB, so that might be it:
https://cloud.google.com/appengine/docs/standard/java/logs/#writing_application_logs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153771335
Since this reporting is getting more complicated (see b/36599833), it'll
be better to have a dedicated class to encapsulate it, which also lets us
keep the tests separate and focus FlowRunner more on its core purpose of
actually running the flow.
Note that this doesn't move the legacy log statement logging because that
specifically must be logged from the FlowRunner.run() method to preserve
the existing log signature matching in our ICANN activity reporting query.
(The new statement is designed to be robust to moves like this since it
doesn't use the logging callsite to match log lines, and it's not in use
yet anyway.)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153762008
This is required by ICANN Consistent Labeling & Display policy that WHOIS domain query response contains registrar abuse contact's phone number and email address. Add a helper function to load registrar contact of a certain type for a given registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153606137
Make it clear that all the user need to do to rectify is to provide a phone number
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153191178
One is an unnecessary import and the other is an incorrectly named
Javadoc parameter.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153095269
We now send PendingActionNotificationResponses in our poll messages upon completion of an asynchronous contact or host deletion. This is part 1 of 2, which begins logging Trid in all enqueued Host/Contact deletion flows for use in batch deletions, and optionally consuming the resultant Trid info to emit a Host/ContactPendingActionNotifcationResponse.
Part 2 will make this response emission non-optional, which will happen once the queue is cleared of all non-Trid containing tasks.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153084197
