These are simply too costly in their current form now that we are handling double-digit QPS, so at a minimum we'd want to refactor these for batched exports using a background thread (like how Stackdriver metrics work). However, upon further review, that work isn't worth doing if this BigQuery table isn't actually being used for anything, and it seems that we aren't using it anymore given that ICANN transaction reporting no longer requires it.
So the simplest thing to do is simply to get rid of this entirely, and just use a combination of Stackdriver metrics and App Engine logs. The eppMetrics BigQuery table is ~1.2 billion rows and takes up 223 GB, so that's not an insignificant GCP billings saving if we can delete it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=215905466
Masks user credentials (tags 'pw' and 'newPW') in EPP XML messages.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207953894
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
We've moved completely to the JSON based reporting framework. The legacy logging statement is only for human consumption, therefore removing the comments. Also fixes a bug where the last argument is not used due to the formatter only expecting 7 arguments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198558998
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
The logging for exceptions in FlowRunner - always at WARNING - has long been sub-optimal. For EppExceptions it's too aggressive/spammy to log at WARNING because it's generally not actionable - EppException gets properly thrown for all kinds of ordinary reasons (trying to create a resource when one already exists with that foreign key) and/or for client misbehavior that we can't control (sending bad parameter values, etc.). For non-EppException RuntimeExceptions, it's redundant with existing logging in EppController.
This CL resolves this by removing that logging in FlowRunner entirely in favor of the EppController logging, where we're now logging EppExceptions at INFO in parallel with the existing logging of RuntimeExceptions at SEVERE. This has the benefit that we're now logging EppExceptions that come from FlowPicker (by way of EppExceptionInProviderException), which previously were unlogged.
Note however that this does mean that in places where we run FlowRunner without EppController - exclusively test code as it stands today - we'd no longer be logging EppExceptions. If that seems like a loss, we could either reinstate logging there (at INFO) and just deal with redundant messages for most EppExceptions, or we could add it manually to places where we call FlowRunner.run() in tests and avoid the redundancy that way.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154733365
This wasn't being recorded correctly because the clientId is only
set in LoginFlow after the flow succeeds, whereas we were
previously logging the clientId before executing the flow. This
adds special handling for LoginFlow. Note that we only set the
metric label to the clientId for valid registrar logins, to ensure
that metric cardinality doesn't grow unbounded (as it might if we
used every arbitrary string passed in as an attempted login).
This also refactors creation and handling of FakesAndMocksModule so
as to be able to make test assertions about EPP metrics from
integration flow tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154048280
Since this feeds into ICANN reporting, we don't want to muddy the data
there with dry-runs, which are always internal-only artifacts of tool usage
and shouldn't really count as real attempts to do SRS actions, since they
are always going to abort with no effect.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153856915
Since this reporting is getting more complicated (see b/36599833), it'll
be better to have a dedicated class to encapsulate it, which also lets us
keep the tests separate and focus FlowRunner more on its core purpose of
actually running the flow.
Note that this doesn't move the legacy log statement logging because that
specifically must be logged from the FlowRunner.run() method to preserve
the existing log signature matching in our ICANN activity reporting query.
(The new statement is designed to be robust to moves like this since it
doesn't use the logging callsite to match log lines, and it's not in use
yet anyway.)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153762008
This fixes recording of number of attempts and command name on EPP
flows, which was broken because a separate metric builder was
being injected in two places, EppController and FlowRunner, with the
one injected into FlowRunner being discarded rather than having changes
applied to the same instance as in EppController.
This also adds a test that the metric is created successfully inside
a flow. Note that tests already exist for EppController to ensure that
the metric is recorded correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152306596
As part of b/36599833, this makes FlowRunner log the appropriate ICANN activity
report field name for each flow it runs as part of a structured JSON log
statement which can be parsed to generate ICANN activity reports (under the key
"icannActivityReportField").
In order to support this, we introduce an annotation for Flow classes called
@ReportingSpec and a corresponding enum of values for this annotation, which is
IcannReportingTypes.ActivityReportField, that stores the mapping of constant
enum values to field names.
The mapping from flows to fields is fairly obvious, with three exceptions:
- Application flows are all accounted under domains, since applications are
technically just deferred domain creates within the EPP protocol
- ClaimsCheckFlow is counted as a domain check
- DomainAllocateFlow is counted as a domain create
In addition, I've added tests to all the corresponding flows that we are
indeed logging what we expect.
We'll also need to log the TLD for this to be useful, but I'm doing that in a
follow-up CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151283411
It was previously only using the name of the inner command XML element,
e.g. "Create", "Delete", "Update", etc. This wasn't very useful because
there was no way to discriminate between operations on different types
of EPP resources.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151131491
This concludes your flow flattening experience. Please
fill out a flow flattening satisfaction survey before
exiting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137903095
This was meant for log replay and has long been ignored/useless.
As part of this, remove execution time from EppResponse since this
was the only thing consuming it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137293734
It is no longer needed since we don't do log-replay and it's
annoying to support in the flat flows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135805088
This allows us to inject an optional once, in FlowRunner, and
inject a non-null value in the flows (not done yet, after this
goes in).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133130485
This change refactors EppMetrics from the mutable self-exporting thing that it
was into a real value type EppMetric, and delegates exporting functionality to the
BigQueryMetricsEnqueuer.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132387660
We've been using the very fragile newline-delimited legacy logging
statement in FlowRunner for ICANN reporting for a long time. While
this is bad in a few ways, the worst is that the parsing of this
logging statement is extremely fragile (e.g. adding/removing fields
can easily break the parsing). This is in fact part of what broke the
ExportLogsServlet parsing last fall ([] and forced us to
recover by manually parsing the log statement (and its XML) in
BigQuery. It also broke again in [] where we were relying
on matching the logging classname, since matching on 'EPP Command'
was considered insufficiently narrow.
This introduces a new JSON-format logging statement to FlowRunner
that fixes both of these problems:
1) it replaces the newline-delimited "format" with a JSON-based
format, so that we can add new fields much more easily and
reliably support logging more structured data
2) it replaces the short 'EPP Command' signature with a much more
targeted 'EPP-REPORTING-LOG-SIGNATURE' signature so that we can
use that alone for matching, rather than relying on the class
name in the log message
What this doesn't fix is the fact that we still need to parse the
XML in BigQuery; we should fix this by logging the parts of the XML
that ICANN reporting needs explicitly, but that'll be a subsequent
change, since while the existing approach is gross, it's actually
much less fragile than just matching the log statement itself.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125902976
Daggerizes all of the EPP flows. This does not change anything yet
about the flows themselves, just how they are invoked, but after
this CL it's safe to @Inject things into flow classes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125382478
The "SessionSource" has nothing to do with sessions (and it's often
used in sessionless contexts). What it does indicate is the endpoint
used to make the request.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125295224
Superuser should only be settable via the tool (see []
which is merged in here but not diffbased, and which removes
the implicit superuser for CharlestonRoad). It is a property
of the request, not of the session (there are no sessions in the tool).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125204707
dryRun is only available via the (sessionless!) tool, and is not
a property of the session.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125203026
TransportCredentials are per-request, not per-session, and
there's no reason to carry them within SessionMetadata.
While I'm in here, get rid of "null" credentials.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125202213
This introduces Actions and Dagger up until FlowRunner. The changes
to the servlets are relatively simple, but the required changes to
the tests, as well as to auxillary EPP endpoints (such as the http
check api and the load test servlet) were vast. I've added some
comments in critique to make the review easier that don't really
make sense as in-code comments for the future.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=124593546
The dark lord Gosling designed the Java package naming system so that
ownership flows from the DNS system. Since we own the domain name
registry.google, it seems only appropriate that we should use
google.registry as our package name.
This change renames directories in preparation for the great package
rename. The repository is now in a broken state because the code
itself hasn't been updated. However this should ensure that git
correctly preserves history for each file.
2016-05-13 18:55:08 -04:00
Renamed from java/com/google/domain/registry/flows/FlowRunner.java (Browse further)
