Commit graph

369 commits

Author SHA1 Message Date
cgoldfeder
b9b2829f7c Log the class names of unimplemented extensions before
throwing the generic user-visible error.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132884249
2016-09-14 15:42:59 -04:00
cgoldfeder
d7443f2eee Fix copy/paste javadoc erroc
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132875203
2016-09-14 15:41:32 -04:00
nickfelt
9dffd64dc7 Cleanup minor things in whitebox metrics code
Specifically:
- remove @RequestScope from EppMetric since it's only for components
- fix to call the better overload of toBigqueryTimestamp
- use the same UUID provider for BigQueryMetricsEnqueuer that already exists for
  the VerifyEntityIntegrityStreamer
- minor cleanup in VerifyEntityIntegrityStreamer (inject projectId vs whole env)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132721794
2016-09-14 15:39:51 -04:00
nickfelt
2537e95de5 Change EppMetric.Builder to use @AutoValue.Builder
Getting rid of builder boilerplate makes my heart sing.  Since we can no
longer @Inject the Builder() constructor, this change adds a provider
in WhiteboxModule that calls a special builderForRequest() factory method,
which gets passed a request ID and Clock and preserves the existing
EppMetric magic that sets the start and end time for you.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132714432
2016-09-14 15:38:22 -04:00
nickfelt
ceb5c2117e Decouple GaeUserCredentials from UserService and simplify tests
This disentangles GaeUserCredentials and UserService, which lets us remove a
bunch of hacky and brittle code from LoginFlowViaConsoleTest.

Previously, GaeUserCredentials was constructed for a user, but then was still
directly calling UserService to check if the user was an admin.  UserService
can be adjusted in tests (via AppEngineRule / LocalServiceTestHelper) but it's
a pain, especially to do dynamically within a single test file.  The hacky
code in LoginFlowViaConsoleTest was working around that restriction.

With this CL, you can pass into GaeUserCredentials whether the user is an
admin or not (for testing) or construct one directly from a UserService object
(for production, and for convenience in tests using an AppEngineRule user).

Note that I also changed EppConsoleAction to @Inject UserService.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132696391
2016-09-14 15:36:53 -04:00
shikhman
bd887e857e Fix incorrect field name in EppMetric
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132688882
2016-09-14 15:35:25 -04:00
nickfelt
cb55ab4f5f Convert straggler GAE modules to basic-scaling
This fixes the proximate cause of b/31380927 and makes alpha frontend usable
again.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132675121
2016-09-14 15:33:57 -04:00
shikhman
aa4ca42cdd Add EPP metrics to flows
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132591518
2016-09-14 15:32:27 -04:00
nickfelt
e478fd09fb Remove straggling JarKeyring link and sort MOE-added imports
Followups from []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132564245
2016-09-14 15:29:30 -04:00
mountford
a63921350b HistoryEntry for extra logic; update fee check
While working on an implementation of TLD-specific logic, it was realized that the extra logic methods would need access to the flow's HistoryEntry, so that things like poll messages could be parented properly.

Also, the update flow had not been fixed to perform the fee check.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132561527
2016-09-14 15:28:03 -04:00
mountford
95cc7ab3d8 Add extra logic for all relevant flows
This CL enhances various domain flows (check, create, delete, renew, restore, transfer, update) so that they invoke the appropriate methods on the object implementing the TLD's RegistryExtraFlowLogic (if any). TldSpecificLogicProxy is also updated to invoke RegistryExtraFlowLogic proxy (if any) to fetch the appropriate price. The tests use a made-up extra flow logic object which can be attached to a test TLD to make sure that the proper routines are being invoked.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132486734
2016-09-14 15:26:36 -04:00
mcilwain
a6db24c8bb Fix NullPointerException in group syncing
We were trying to set the scopes too early in the creation of a
GoogleCredential.  They aren't yet known at this point in the code and should be
an empty collection (and also not null, which yields an NPE).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132477868
2016-09-14 15:25:07 -04:00
mcilwain
2b42118aaf Make boolean operator precedence unambiguous in ClaimsListShard
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132442796
2016-09-14 15:19:15 -04:00
mcilwain
8cfa643598 Switch Optional.ofNullable() to Optional.fromNullable() in EppMetric
The latter is the canonical way to call it; the former is not available in
public Guava (and is just a pass-through to fromNullable anyway).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132441956
2016-09-14 15:17:49 -04:00
mcilwain
cadf9d4af2 Use smaller shard size in ClaimsListShardTest
The default production value of 10,000 was unnecessarily large for testing
purposes.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132441792
2016-09-14 15:16:22 -04:00
shikhman
4652688585 Add default DistributionFitter implementation to EventMetric
This DistributionFitter is suitable for tracking the latency of network calls.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132400538
2016-09-07 12:20:46 -04:00
shikhman
42a39b0ddc Refactor EppMetrics into the EppMetric value type
This change refactors EppMetrics from the mutable self-exporting thing that it
was into a real value type EppMetric, and delegates exporting functionality to the
BigQueryMetricsEnqueuer.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132387660
2016-09-07 12:19:22 -04:00
jart
b77ebd1df9 Don't use modules to instantiate classes
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132380251
2016-09-07 12:17:58 -04:00
mcilwain
3b4b7a475b Replace VoidKeyring with InMemoryKeyring and a dummy module
VoidKeyring always threw exceptions whenever any of its methods were called,
which caused several parts of the system to fail early (and thus required a
proper Keyring to be implemented almost immediately, early on in the "just
playing around with the system" phase).

I'm swapping this out with an InMemoryKeyring which is supplied by
DummyKeyringModule, which, instead of throwing exceptions, returns dummy
values, delaying the onset of errors to when connecting to external services
is attempted.  This pushes off the required implementation of a real Keyring-
providing module, allowing the system to be played around with more first.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132361157
2016-09-07 12:15:08 -04:00
mcilwain
fea8c96f79 Remove GoogleCredential method call that isn't publicly released yet
It turns out that we don't actually need it for the system to function normally.
When the next version of the OAuth2 API is released and this is included, I'll
add the call back in.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132359891
2016-09-07 12:12:16 -04:00
shikhman
e5a0392994 Make StackdriverWriter#flush() return early when attempting to flush empty
timeseries

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132359115
2016-09-07 12:10:49 -04:00
shikhman
0e8d9e3859 Force endTimestamp to be strictly greater than or equal to startTimestamp
This fixes an error that we've been seeing from Stackdriver.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132358790
2016-09-07 12:09:23 -04:00
ctingue
daca7d65c2 Allow PasswordGenerator to use different alphabets
Per mcilwain's suggestion in the LRP design doc, LRP tokens should use a Base58 alphabet. I'll move PasswordGenerator out of the tools package and into a utils class in a future CL, as we'll want to use this generator in the LrpToken class itself rather than relegate the token definition to a tool.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132358363
2016-09-07 12:07:58 -04:00
shikhman
dbb9ef80c5 Refactor ConcurrentHashMap creation code to a helper method
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132352263
2016-09-07 12:06:26 -04:00
shikhman
0538c1e210 Add method to create and register EventMetrics
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132346375
2016-09-07 12:03:28 -04:00
shikhman
2ba9b01a13 Add the EventMetric metric type
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132345599
2016-09-07 12:01:58 -04:00
jianglai
6641f105b7 Create a separate billing event when EAP is applied
When EAP is involed we current have one billing event for domain create that
has the create fee and EAP fee lumped together. Change it to record two
separate billing events for each.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132335349
2016-09-07 12:00:28 -04:00
shikhman
969d9483ae Add encoding for the Stackdriver Distribution value type to StackdriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132327732
2016-09-07 11:58:58 -04:00
ctingue
6f241c5605 Index redemptionHistoryEntry on LrpToken
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132323443
2016-09-07 11:57:30 -04:00
shikhman
93bf68039b Add lock.unlock() to finally clause in Counter
It is good practice to unlock a lock in a finally clause.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132302413
2016-09-07 11:56:00 -04:00
shikhman
dcb189943b Add the Distribution data type for instrumentation
This is one of a series of CLs adding a new metric type, EventMetric, which
is used for tracking numerical distributions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132103552
2016-09-07 11:54:26 -04:00
shikhman
180240ae04 Wire in MetricsReporter to FrontendServlet
This work is identical to the work done for BackendServlet in
[]

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132100448
2016-09-07 11:52:59 -04:00
ctingue
c75ab37dab Export LrpToken to BigQuery
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132083818
2016-09-07 11:51:28 -04:00
mcilwain
07135f6190 Don't allow null in BillingEvent.setFlags()
It's better that it always takes a non-null ImmutableSet, which may either be
empty or contain elements.  That way the ugliness of nullness is contained
just to the entity class itself, and all other code that interacts with it
can always be assured of having a real set to deal with.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132066238
2016-09-02 13:57:50 -04:00
mcilwain
c06133435b Consolidate some cron groups and use fanout action for more retrying
This is the first step in consolidating our task queues down into a smaller
number.  We have lots of tasks that run quite infrequently, and they can all run
in the same queue to get retry semantics without needing a new queue for each
cron entry.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131990472
2016-09-02 13:56:23 -04:00
shikhman
7d1a30d8d6 Remove backfill for DnsWriter and PricingEngine
The backfill for these is no longer necessary as all of the Registry entities in all environments have been backfilled.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131984138
2016-09-02 13:53:28 -04:00
cgoldfeder
5098b03af4 DeReference the codebase
This change replaces all Ref objects in the code with Key objects. These are
stored in datastore as the same object (raw datastore keys), so this is not
a model change.

Our best practices doc says to use Keys not Refs because:
 * The .get() method obscures what's actually going on
   - Much harder to visually audit the code for datastore loads
   - Hard to distinguish Ref<T> get()'s from Optional get()'s and Supplier get()'s
 * Implicit ofy().load() offers much less control
   - Antipattern for ultimate goal of making Ofy injectable
   - Can't control cache use or batch loading without making ofy() explicit anyway
 * Serialization behavior is surprising and could be quite dangerous/incorrect
   - Can lead to serialization errors. If it actually worked "as intended",
     it would lead to a Ref<> on a serialized object being replaced upon
     deserialization with a stale copy of the old value, which could potentially
     break all kinds of transactional expectations
 * Having both Ref<T> and Key<T> introduces extra boilerplate everywhere
   - E.g. helper methods all need to have Ref and Key overloads, or you need to
     call .key() to get the Key<T> for every Ref<T> you want to pass in
   - Creating a Ref<T> is more cumbersome, since it doesn't have all the create()
     overloads that Key<T> has, only create(Key<T>) and create(Entity) - no way to
     create directly from kind+ID/name, raw Key, websafe key string, etc.

(Note that Refs are treated specially by Objectify's @Load method and Keys are not;
we don't use that feature, but it is the one advantage Refs have over Keys.)

The direct impetus for this change is that I am trying to audit our use of memcache,
and the implicit .get() calls to datastore were making that very hard.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131965491
2016-09-02 13:50:20 -04:00
shikhman
1a60073b24 Small fixes in StackdriverModule
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131955504
2016-09-02 13:48:47 -04:00
shikhman
c11ac3129f Factor out labelValue length check to abstract base class and small name fix
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131902964
2016-09-02 13:47:21 -04:00
mcilwain
a0f1a8b0bc Copy two more fields when building delegated admin GoogleCredential
The absence of the service account scopes was causing a NullPointerException
when the Google groups syncing action was running.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131870610
2016-09-02 13:45:52 -04:00
shikhman
c309560892 Rename fields in EppMetrics and EntityIntegrityAlertsSchema
This is one of a series of CLs which will refactor EppMetrics into a value type
and Metrics into a stateless class which will have an export(EppMetrics requestDetails)
method to export EPP metrics in a stateless way. Once EppMetrics is a value
type, I will create a new StackdriverEppMetrics that will also accept the value
type via an incrementRequests(EppMetrics requestDetails), allowing us to
monitor EPP via BigQuery and Stackdriver with minimum code duplication.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131711810
2016-08-30 14:18:07 -04:00
mountford
59ac00478e Improve BaseFee class and its children
Checks are added to ensure that fees are always non-negative, while credits are always negative, as required by the EPP fee extension specification. Also, BaseFee is made specifically abstract.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131620110
2016-08-30 14:16:41 -04:00
cgoldfeder
8059ab5c90 A @DoNotHydrate annotation for toHydratedString
This is needed for a soon-to-be-submitted CL that changes
all Refs to Keys and therefore removes the logic in
toHydratedString that doesn't expand Keys. We need to be
able to tag types as unexpandable to avoid cycles. It
would be better to tag fields, not types, but that is a
much harder change and not currently needed by the use
case of the following CL, so for now this suffices.

While I am in here, add tests for all of the features
of toHydratedString.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131618634
2016-08-30 14:15:16 -04:00
shikhman
d77dced024 Fix NPE in StackdriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131603566
2016-08-30 14:13:50 -04:00
ctingue
942dc58251 Add valid TLD check to LrpToken validation
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131438274
2016-08-30 14:10:54 -04:00
mcilwain
db8f8d5ba4 Mark rest of Domain Registry for App Engine compatibility
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131296453
2016-08-26 09:48:37 -04:00
mcilwain
9bdb2ef1f3 Make Stackdriver tests conform to testing practices in rest of codebase
The major changes are using the ExpectedException rule instead of a try/catch
pattern, asserting the message for thrown exceptions, defaulting to JUnit4
unless Mockito is really necessary, and making each test examine one behavior.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131211346
2016-08-26 09:45:10 -04:00
shikhman
1c5b4e16c6 Add metrics to PublishDnsUpdatesAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131190862
2016-08-26 09:43:30 -04:00
mcilwain
52bb8a5773 Reduce default [] shards to 100
We've continuously had concurrent modification exceptions for our regularly
occurring []s that run on thousands of shards, perhaps unnecessarily so.
These exceptions started after the last major [] framework refactoring,
which changed the default number of shards from 100 to essentially infinite.  I
don't think infinite is the way to go, and 100 shards should be more than
sufficient for anything we're currently running.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131175353
2016-08-26 09:40:17 -04:00
shikhman
e6beba2d39 Change resource type for Stackdriver to fix metrics push
Currently, it's forbidden for custom metrics to write to the gae_app resource
type.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131076789
2016-08-23 15:48:49 -04:00