Commit graph

79 commits

Author SHA1 Message Date
jianglai
ceadfea11f Remove @Inject annotation in StackDriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170111976
2017-10-04 16:16:45 -04:00
shikhman
f27dd70c9e Automated g4 rollback of changelist 169922096.
*** Reason for rollback ***

Rolling back to fix []
*** Original change description ***

Remove @Inject annotation in StackDriverWriter

StackDriverWriter is provided by java.google.registry.monitoring.whitebox.StackdriverModule. The @Inject annotation in its constructor is not used. It also uses outdated named binding like @Named(stackdriverGcpProject) which causes confusion because that name is not provided anywhere in the code.

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169931047
2017-10-04 16:16:45 -04:00
jianglai
2814561e92 Remove @Inject annotation in StackDriverWriter
StackDriverWriter is provided by java.google.registry.monitoring.whitebox.StackdriverModule. The @Inject annotation in its constructor is not used. It also uses outdated named binding like @Named(stackdriverGcpProject) which causes confusion because that name is not provided anywhere in the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169922096
2017-10-04 16:16:45 -04:00
guyben
fa70bd272a Add tool to check if other requests are still running
This is needed for the Lock.java enhancement where a lock will be implicitly
released if the request owning it dies.

No matter which solution we want for refactoring the Lock itself, we will need this class.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167600314
2017-09-12 15:51:50 -04:00
guyben
e224a67eda Change @Auth to an AutoValue, and created a set of predefined Auths
We want to be safer and more explicit about the authentication needed by the many actions that exist.

As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
2017-08-01 16:33:10 -04:00
mcilwain
0fdde1cc9e Change FibonacciFitter to just be a utility method
It turns out that StackdriverWriter uses reflection on the class of
the DistributionFitter instance, so rather than giving it custom
handling for FibonacciFitters, it's easier to turn FibonacciFitter
into a single utility method that takes advantage of existing support
for CustomFitters.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158190147
2017-06-14 10:31:29 -04:00
mcilwain
bb67841884 Add metrics for async batch operation processing
We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:

* An incrementable metric for each async task processed (split out by
  type of task and result).
* Two event metrics for processing time between when a task is enqueued
  and when it is processed -- tracked separately for contact/host
  deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
  run (this is usually 0). Tracked separately for contact/host deletion
  and DNS refresh on host rename.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
2017-06-05 18:17:09 -04:00
mcilwain
1adeb57fea Add a Fibonacci fitter for metrics bucketing
A Fibonacci fitter is useful in situations where you want more precision on the
low end than an ExponentialFitter with exponent base 2 provides without the
hassle of dealing with non-integer boundaries, such as would be created by an
exponential fitter with a base of less than 2. Fibonacci fitters are ideal for
integer metrics that are bounded across a certain range, e.g. integers between 1
and 1,000.

This also cleans up some unit test comments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156773367
2017-05-23 17:22:49 -04:00
shikhman
413a634533 Automated g4 rollback of changelist 156106183.
*** Reason for rollback ***

This change is crashing alpha

*** Original change description ***

Change metrics ThreadFactory to produce daemon threads

***

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156336467
2017-05-17 15:19:10 -04:00
shikhman
e1f4df86bd Change metrics ThreadFactory to produce daemon threads
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156106183
2017-05-17 12:21:12 -04:00
mcilwain
4260fb573f Use the actual EPP command flow name for EppMetrics
It was previously only using the name of the inner command XML element,
e.g. "Create", "Delete", "Update", etc. This wasn't very useful because
there was no way to discriminate between operations on different types
of EPP resources.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151131491
2017-03-27 13:30:05 -04:00
mcilwain
3a13a4d6b6 Add monitoring/metrics/contrib package to Nomulus release
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150007228
2017-03-21 14:59:31 -04:00
Ben McIlwain
0cbc2bdab6 Use Immutable types for static final fields
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149260759
2017-03-07 13:53:46 -05:00
mountford
9a15f08b3a Add Truth subjects for incrementable and event metrics
This CL adds Truth framework subjects to some metrics in the Stackdriver metrics library, in a contrib subpackage. It doesn't deal with gauge metrics, and for event metrics, the assertions can only be that a metric has or does not have a distribution for a particular set of label values. Asserting more fine-grained propositions regarding the distribution will require a distribution subject.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149112692
2017-03-07 13:46:38 -05:00
mcilwain
d6e6e8a49c Re-add some accidentally removed license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146694719
2017-02-06 16:45:23 -05:00
mmuller
b70f57b7c7 Update copyright year on all license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146111211
2017-02-02 16:27:22 -05:00
mcilwain
813f837956 Re-add Stackdriver metrics code
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146000275
2017-01-30 15:03:53 -05:00
shikhman
3f9c53b850 Refactor Stackdriver-over-REST specific code to a different package
Downstream users who use gRPC rather than REST don't want to pull down
rest-related dependencies.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145834701
2017-01-30 15:03:53 -05:00
mcilwain
8f8d67a17b Inject the BigQuery streaming metrics queue
This makes local development easier by allowing a stub to be injected for the
queue.

Based on Justin Graham's PR: https://github.com/google/nomulus/pull/51

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=144439454
2017-01-13 10:29:36 -05:00
shikhman
e981bad0b3 Rename all encoding methods to start with the word encode for consistency
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143973446
2017-01-12 13:58:31 -05:00
mcilwain
eaec03e670 Move ConfigModule and LocalTestConfig into RegistryConfig
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
2017-01-09 12:01:09 -05:00
jart
734130aa73 Restructure Maven dependencies in build
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072

We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018

A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55

The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.

Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:

- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
2017-01-09 11:59:04 -05:00
mcilwain
28f6c770c8 Add MOE equivalence for sync on 2016-12-19
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142449539
2017-01-09 11:59:04 -05:00
mountford
0afafeff03 Move VerifyEntityIntegrityAction to batch package
By moving []s into the batch package, which is not included in the frontend service, we pave the way to remove the dependency of frontend on the [] library.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142265351
2016-12-19 11:09:20 -05:00
mcilwain
2b7d580bb3 Run buildifier on codebase to format BUILD files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140362453
2016-11-28 18:15:21 -05:00
cgoldfeder
84009eaccb Scope down TransferData to only ContactResource and DomainResource
HostResource and DomainApplication are not transferable, (or at
least, not directly in the case of hosts) and have no need for
the TransferData field. In a flat-flow world, we can push it down
to where it's actually used.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=139201423
2016-11-15 13:47:28 -05:00
jianglai
59d998954c Use correct <a> tag syntax in javadoc @see tag
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137946021
2016-11-02 15:19:34 -04:00
jart
2e81de9954 Make essential Bazel packages publicly visible
This allows separate Bazel projects to reference Nomulus as an external
repository. They can then copy the []
directory structure into their own project and customize the Action
and Module lists for the GAE modules in their own deployment.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136863886
2016-10-24 11:57:00 -04:00
jianglai
cbe6d2b520 Resolve some deprecation warnings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136399461
2016-10-18 10:32:47 -04:00
ctingue
67da55536f Clean up misc Javadoc and deprecation warnings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136156922
2016-10-14 17:00:33 -04:00
shikhman
f76bc70f91 Preserve test logs and test summary output for Kokoro CI runs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135494972
2016-10-14 16:57:43 -04:00
mcilwain
b65b855067 Always use the constructor to make Immutable Collection Builders
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135359669
2016-10-07 14:37:02 -04:00
shikhman
e19546ffb4 Fix encoding error for Stackdriver metrics of cumulative points
The Stackdriver API requires that the end time always be greater than the start
time for cumulative metric points.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134396192
2016-10-03 15:43:16 -04:00
mcilwain
ffcbdbf37b Set method to POST on actions called from cron fanout
The default is to support GET, which doesn't work with cron fanout which only
uses POST.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134284855
2016-09-26 13:50:03 -04:00
nickfelt
9dffd64dc7 Cleanup minor things in whitebox metrics code
Specifically:
- remove @RequestScope from EppMetric since it's only for components
- fix to call the better overload of toBigqueryTimestamp
- use the same UUID provider for BigQueryMetricsEnqueuer that already exists for
  the VerifyEntityIntegrityStreamer
- minor cleanup in VerifyEntityIntegrityStreamer (inject projectId vs whole env)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132721794
2016-09-14 15:39:51 -04:00
nickfelt
2537e95de5 Change EppMetric.Builder to use @AutoValue.Builder
Getting rid of builder boilerplate makes my heart sing.  Since we can no
longer @Inject the Builder() constructor, this change adds a provider
in WhiteboxModule that calls a special builderForRequest() factory method,
which gets passed a request ID and Clock and preserves the existing
EppMetric magic that sets the start and end time for you.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132714432
2016-09-14 15:38:22 -04:00
shikhman
bd887e857e Fix incorrect field name in EppMetric
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132688882
2016-09-14 15:35:25 -04:00
mcilwain
8cfa643598 Switch Optional.ofNullable() to Optional.fromNullable() in EppMetric
The latter is the canonical way to call it; the former is not available in
public Guava (and is just a pass-through to fromNullable anyway).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132441956
2016-09-14 15:17:49 -04:00
shikhman
4652688585 Add default DistributionFitter implementation to EventMetric
This DistributionFitter is suitable for tracking the latency of network calls.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132400538
2016-09-07 12:20:46 -04:00
shikhman
42a39b0ddc Refactor EppMetrics into the EppMetric value type
This change refactors EppMetrics from the mutable self-exporting thing that it
was into a real value type EppMetric, and delegates exporting functionality to the
BigQueryMetricsEnqueuer.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132387660
2016-09-07 12:19:22 -04:00
shikhman
e5a0392994 Make StackdriverWriter#flush() return early when attempting to flush empty
timeseries

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132359115
2016-09-07 12:10:49 -04:00
shikhman
0e8d9e3859 Force endTimestamp to be strictly greater than or equal to startTimestamp
This fixes an error that we've been seeing from Stackdriver.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132358790
2016-09-07 12:09:23 -04:00
shikhman
dbb9ef80c5 Refactor ConcurrentHashMap creation code to a helper method
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132352263
2016-09-07 12:06:26 -04:00
shikhman
0538c1e210 Add method to create and register EventMetrics
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132346375
2016-09-07 12:03:28 -04:00
shikhman
2ba9b01a13 Add the EventMetric metric type
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132345599
2016-09-07 12:01:58 -04:00
shikhman
969d9483ae Add encoding for the Stackdriver Distribution value type to StackdriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132327732
2016-09-07 11:58:58 -04:00
shikhman
93bf68039b Add lock.unlock() to finally clause in Counter
It is good practice to unlock a lock in a finally clause.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132302413
2016-09-07 11:56:00 -04:00
shikhman
dcb189943b Add the Distribution data type for instrumentation
This is one of a series of CLs adding a new metric type, EventMetric, which
is used for tracking numerical distributions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132103552
2016-09-07 11:54:26 -04:00
cgoldfeder
5098b03af4 DeReference the codebase
This change replaces all Ref objects in the code with Key objects. These are
stored in datastore as the same object (raw datastore keys), so this is not
a model change.

Our best practices doc says to use Keys not Refs because:
 * The .get() method obscures what's actually going on
   - Much harder to visually audit the code for datastore loads
   - Hard to distinguish Ref<T> get()'s from Optional get()'s and Supplier get()'s
 * Implicit ofy().load() offers much less control
   - Antipattern for ultimate goal of making Ofy injectable
   - Can't control cache use or batch loading without making ofy() explicit anyway
 * Serialization behavior is surprising and could be quite dangerous/incorrect
   - Can lead to serialization errors. If it actually worked "as intended",
     it would lead to a Ref<> on a serialized object being replaced upon
     deserialization with a stale copy of the old value, which could potentially
     break all kinds of transactional expectations
 * Having both Ref<T> and Key<T> introduces extra boilerplate everywhere
   - E.g. helper methods all need to have Ref and Key overloads, or you need to
     call .key() to get the Key<T> for every Ref<T> you want to pass in
   - Creating a Ref<T> is more cumbersome, since it doesn't have all the create()
     overloads that Key<T> has, only create(Key<T>) and create(Entity) - no way to
     create directly from kind+ID/name, raw Key, websafe key string, etc.

(Note that Refs are treated specially by Objectify's @Load method and Keys are not;
we don't use that feature, but it is the one advantage Refs have over Keys.)

The direct impetus for this change is that I am trying to audit our use of memcache,
and the implicit .get() calls to datastore were making that very hard.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131965491
2016-09-02 13:50:20 -04:00
shikhman
1a60073b24 Small fixes in StackdriverModule
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131955504
2016-09-02 13:48:47 -04:00