Commit graph

494 commits

Author SHA1 Message Date
shikhman
2ba9b01a13 Add the EventMetric metric type
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132345599
2016-09-07 12:01:58 -04:00
jianglai
6641f105b7 Create a separate billing event when EAP is applied
When EAP is involed we current have one billing event for domain create that
has the create fee and EAP fee lumped together. Change it to record two
separate billing events for each.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132335349
2016-09-07 12:00:28 -04:00
shikhman
969d9483ae Add encoding for the Stackdriver Distribution value type to StackdriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132327732
2016-09-07 11:58:58 -04:00
ctingue
6f241c5605 Index redemptionHistoryEntry on LrpToken
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132323443
2016-09-07 11:57:30 -04:00
shikhman
93bf68039b Add lock.unlock() to finally clause in Counter
It is good practice to unlock a lock in a finally clause.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132302413
2016-09-07 11:56:00 -04:00
shikhman
dcb189943b Add the Distribution data type for instrumentation
This is one of a series of CLs adding a new metric type, EventMetric, which
is used for tracking numerical distributions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132103552
2016-09-07 11:54:26 -04:00
shikhman
180240ae04 Wire in MetricsReporter to FrontendServlet
This work is identical to the work done for BackendServlet in
[]

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132100448
2016-09-07 11:52:59 -04:00
ctingue
c75ab37dab Export LrpToken to BigQuery
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132083818
2016-09-07 11:51:28 -04:00
mcilwain
07135f6190 Don't allow null in BillingEvent.setFlags()
It's better that it always takes a non-null ImmutableSet, which may either be
empty or contain elements.  That way the ugliness of nullness is contained
just to the entity class itself, and all other code that interacts with it
can always be assured of having a real set to deal with.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132066238
2016-09-02 13:57:50 -04:00
mcilwain
c06133435b Consolidate some cron groups and use fanout action for more retrying
This is the first step in consolidating our task queues down into a smaller
number.  We have lots of tasks that run quite infrequently, and they can all run
in the same queue to get retry semantics without needing a new queue for each
cron entry.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131990472
2016-09-02 13:56:23 -04:00
shikhman
7d1a30d8d6 Remove backfill for DnsWriter and PricingEngine
The backfill for these is no longer necessary as all of the Registry entities in all environments have been backfilled.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131984138
2016-09-02 13:53:28 -04:00
cgoldfeder
5098b03af4 DeReference the codebase
This change replaces all Ref objects in the code with Key objects. These are
stored in datastore as the same object (raw datastore keys), so this is not
a model change.

Our best practices doc says to use Keys not Refs because:
 * The .get() method obscures what's actually going on
   - Much harder to visually audit the code for datastore loads
   - Hard to distinguish Ref<T> get()'s from Optional get()'s and Supplier get()'s
 * Implicit ofy().load() offers much less control
   - Antipattern for ultimate goal of making Ofy injectable
   - Can't control cache use or batch loading without making ofy() explicit anyway
 * Serialization behavior is surprising and could be quite dangerous/incorrect
   - Can lead to serialization errors. If it actually worked "as intended",
     it would lead to a Ref<> on a serialized object being replaced upon
     deserialization with a stale copy of the old value, which could potentially
     break all kinds of transactional expectations
 * Having both Ref<T> and Key<T> introduces extra boilerplate everywhere
   - E.g. helper methods all need to have Ref and Key overloads, or you need to
     call .key() to get the Key<T> for every Ref<T> you want to pass in
   - Creating a Ref<T> is more cumbersome, since it doesn't have all the create()
     overloads that Key<T> has, only create(Key<T>) and create(Entity) - no way to
     create directly from kind+ID/name, raw Key, websafe key string, etc.

(Note that Refs are treated specially by Objectify's @Load method and Keys are not;
we don't use that feature, but it is the one advantage Refs have over Keys.)

The direct impetus for this change is that I am trying to audit our use of memcache,
and the implicit .get() calls to datastore were making that very hard.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131965491
2016-09-02 13:50:20 -04:00
shikhman
1a60073b24 Small fixes in StackdriverModule
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131955504
2016-09-02 13:48:47 -04:00
shikhman
c11ac3129f Factor out labelValue length check to abstract base class and small name fix
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131902964
2016-09-02 13:47:21 -04:00
mcilwain
a0f1a8b0bc Copy two more fields when building delegated admin GoogleCredential
The absence of the service account scopes was causing a NullPointerException
when the Google groups syncing action was running.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131870610
2016-09-02 13:45:52 -04:00
shikhman
c309560892 Rename fields in EppMetrics and EntityIntegrityAlertsSchema
This is one of a series of CLs which will refactor EppMetrics into a value type
and Metrics into a stateless class which will have an export(EppMetrics requestDetails)
method to export EPP metrics in a stateless way. Once EppMetrics is a value
type, I will create a new StackdriverEppMetrics that will also accept the value
type via an incrementRequests(EppMetrics requestDetails), allowing us to
monitor EPP via BigQuery and Stackdriver with minimum code duplication.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131711810
2016-08-30 14:18:07 -04:00
mountford
59ac00478e Improve BaseFee class and its children
Checks are added to ensure that fees are always non-negative, while credits are always negative, as required by the EPP fee extension specification. Also, BaseFee is made specifically abstract.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131620110
2016-08-30 14:16:41 -04:00
cgoldfeder
8059ab5c90 A @DoNotHydrate annotation for toHydratedString
This is needed for a soon-to-be-submitted CL that changes
all Refs to Keys and therefore removes the logic in
toHydratedString that doesn't expand Keys. We need to be
able to tag types as unexpandable to avoid cycles. It
would be better to tag fields, not types, but that is a
much harder change and not currently needed by the use
case of the following CL, so for now this suffices.

While I am in here, add tests for all of the features
of toHydratedString.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131618634
2016-08-30 14:15:16 -04:00
shikhman
d77dced024 Fix NPE in StackdriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131603566
2016-08-30 14:13:50 -04:00
ctingue
942dc58251 Add valid TLD check to LrpToken validation
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131438274
2016-08-30 14:10:54 -04:00
mcilwain
db8f8d5ba4 Mark rest of Domain Registry for App Engine compatibility
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131296453
2016-08-26 09:48:37 -04:00
mcilwain
9bdb2ef1f3 Make Stackdriver tests conform to testing practices in rest of codebase
The major changes are using the ExpectedException rule instead of a try/catch
pattern, asserting the message for thrown exceptions, defaulting to JUnit4
unless Mockito is really necessary, and making each test examine one behavior.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131211346
2016-08-26 09:45:10 -04:00
shikhman
1c5b4e16c6 Add metrics to PublishDnsUpdatesAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131190862
2016-08-26 09:43:30 -04:00
mcilwain
52bb8a5773 Reduce default [] shards to 100
We've continuously had concurrent modification exceptions for our regularly
occurring []s that run on thousands of shards, perhaps unnecessarily so.
These exceptions started after the last major [] framework refactoring,
which changed the default number of shards from 100 to essentially infinite.  I
don't think infinite is the way to go, and 100 shards should be more than
sufficient for anything we're currently running.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131175353
2016-08-26 09:40:17 -04:00
shikhman
e6beba2d39 Change resource type for Stackdriver to fix metrics push
Currently, it's forbidden for custom metrics to write to the gae_app resource
type.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131076789
2016-08-23 15:48:49 -04:00
shikhman
87314c4469 Add missing dependency to open source build
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130981257
2016-08-22 17:38:40 -04:00
shikhman
b04e54994e Fix open source build
Add missing dependencies to open source build.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130978680
2016-08-22 17:37:01 -04:00
mcilwain
dd09dc1837 Rename soy templates to use consistent naming scheme
We use {EPP resource}{action} as the naming scheme everywhere else, so we shoudl do so here too.  It's generally nicer for files to group by type of EPP resource (so all of the domain actions are together) than by type of action.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130955192
2016-08-22 14:48:40 -04:00
mcilwain
965cf6edc3 Add registry_tool host_create command
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130953679
2016-08-22 14:45:54 -04:00
shikhman
91f8b6da38 Change to metrics to keep track of when the metric value was first set
This CL also adds IncrementableMetric#reset() methods to allow resetting the
value and start timestamp of IncrementableMetrics.

This is necessary because some backends, like Stackdriver, use non-monotonic
changes in cumulative metric values to detect timeseries restarts. Tracking and
re-setting the start timestamp allows users to track mostly monotonic metrics
which may have non-monotonic discontinuities.

See https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TimeSeries#Point for
more details.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130795229
2016-08-22 14:44:28 -04:00
shikhman
b6eaba08eb Improve service handling and add unit tests for threading behavior in MetricExporter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130790170
2016-08-22 14:43:05 -04:00
nickfelt
6915e35800 Remove unused redirectIfNotLoggedIn() method
It's superseded by RequestHandler's processing of @Action(requireLogin = true), and is no longer used anywhere:
[]

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130788873
2016-08-22 14:41:41 -04:00
ctingue
61bd6159c9 Log-and-swallow post-kickoff in DeleteEppResource
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130787799
2016-08-22 14:40:14 -04:00
cgoldfeder
0e4ce70610 Fix broken JavaDoc
@see tags can't contain a label and a link.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130783416
2016-08-22 14:38:46 -04:00
cgoldfeder
d25be65ed8 Remove an unused import and change @link to @code
...to stop the warnings in this file.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130759383
2016-08-22 14:35:57 -04:00
ctingue
1894b2308b Handle LRP tokens in flows
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130679951
2016-08-22 14:30:19 -04:00
cgoldfeder
5ff8b9377c Remove unused request parameter extractor
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130672143
2016-08-22 14:24:44 -04:00
shikhman
f10a7d8fb0 Enforce monotonicity for IncrementableMetrics
This change enforces that IncrementableMetrics should only monotonically
increase in value, and adds a new increment() method to increment by one, which is slightly faster than incrementBy (due to a lack of non-negative checking) in the common case that the counter should only be incremented by one.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130578421
2016-08-22 14:20:33 -04:00
shikhman
01e38790fd Fix visibility on MetricsReporter#startMetricExporter()
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130576851
2016-08-22 14:19:06 -04:00
shikhman
7a041d066f Improve StackdriverWriter code
Add additional logic to handle cases when a MetricDescriptor is already defined
on the server, fix an NPE in related code, and add additional tests to ensure
that TimeSeries created from native MetricPoints are well-formed.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130555708
2016-08-22 14:17:38 -04:00
mcilwain
9e810c4b36 Add gtech_tool target to tools BUILD
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130545312
2016-08-22 14:16:12 -04:00
shikhman
564bbdef83 Enforce very basic URL-like metric names
This change enforces URL-like metric names to match the behavior of
Stackdriver. The StackdriverWriter no longer prepends a slash, which was a
crutch.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130537347
2016-08-22 14:11:58 -04:00
shikhman
b45c2ca2ee Wire in MetricReporter into the backend servlet to publish metrics
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130468842
2016-08-22 14:07:45 -04:00
mountford
4a34807b1d RDAP: Use IANA identifier as the registrar handle
According to the ICAAN operation profile:

2.1.7. Registries MUST support lookup for entities with the registrar role within other objects using the handle (as described in 3.1.5 of RFC7482). The handle of the entity with the registrar role MUST be equal to IANA Registrar ID. The entity with the registrar role in the RDAP response MUST contain a publicIDs member to identify the IANA Registrar ID from the IANA’s Registrar ID registry. The type value of the publicID object MUST be equal to IANA Registrar ID.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130452501
2016-08-22 14:06:07 -04:00
jianglai
27820c512e Small fixes for orphan glue cleanup []
Moving to tools module for consistency.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130435371
2016-08-22 14:04:44 -04:00
shikhman
9b4ef23d92 Change project to projectResource to better match Stackdriver sample code
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130432024
2016-08-22 14:01:50 -04:00
jianglai
d2f033d9ed Ad-hoc refresh of all domains to clean up orphan glues
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130424917
2016-08-22 13:57:22 -04:00
mcilwain
467286d79e Remove registrar certificate state dead code
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130418304
2016-08-22 13:55:54 -04:00
mountford
0e794c790a Add additional methods for manipulating timed transition maps
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130311938
2016-08-15 17:13:29 -04:00
nickfelt
d1ea3e3a68 Fix NPE bug in DomainCommand.cloneAndLinkReferences()
See b/30806813 for more context.  Copied from there:  This appears to be happening if we get an EPP domain create command that is missing any contacts (but has a registrant; with no registrant we exercise a different codepath).  In that case, JAXB leaves the contacts field on the Create null, and we try to pass it into Sets.union() as a result of Corey's refactoring in [] that changed contact loading to load the contacts and registrant all at once.  The fix is just to apply nullToEmpty() first.

Note that it's always an error to try to create a domain without any non-registrant contacts, but that's supposed to happen later on in BaseDomainCreateFlow.verifyCreateIsAllowed() via validateRequiredContactsPresent(), which will produce a nice error message.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130309019
2016-08-15 17:13:29 -04:00