Commit graph

42 commits

Author SHA1 Message Date
mcilwain
0fdde1cc9e Change FibonacciFitter to just be a utility method
It turns out that StackdriverWriter uses reflection on the class of
the DistributionFitter instance, so rather than giving it custom
handling for FibonacciFitters, it's easier to turn FibonacciFitter
into a single utility method that takes advantage of existing support
for CustomFitters.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158190147
2017-06-14 10:31:29 -04:00
mcilwain
bb67841884 Add metrics for async batch operation processing
We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:

* An incrementable metric for each async task processed (split out by
  type of task and result).
* Two event metrics for processing time between when a task is enqueued
  and when it is processed -- tracked separately for contact/host
  deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
  run (this is usually 0). Tracked separately for contact/host deletion
  and DNS refresh on host rename.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
2017-06-05 18:17:09 -04:00
mcilwain
1adeb57fea Add a Fibonacci fitter for metrics bucketing
A Fibonacci fitter is useful in situations where you want more precision on the
low end than an ExponentialFitter with exponent base 2 provides without the
hassle of dealing with non-integer boundaries, such as would be created by an
exponential fitter with a base of less than 2. Fibonacci fitters are ideal for
integer metrics that are bounded across a certain range, e.g. integers between 1
and 1,000.

This also cleans up some unit test comments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156773367
2017-05-23 17:22:49 -04:00
mcilwain
3a13a4d6b6 Add monitoring/metrics/contrib package to Nomulus release
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=150007228
2017-03-21 14:59:31 -04:00
mountford
9a15f08b3a Add Truth subjects for incrementable and event metrics
This CL adds Truth framework subjects to some metrics in the Stackdriver metrics library, in a contrib subpackage. It doesn't deal with gauge metrics, and for event metrics, the assertions can only be that a metric has or does not have a distribution for a particular set of label values. Asserting more fine-grained propositions regarding the distribution will require a distribution subject.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149112692
2017-03-07 13:46:38 -05:00
mcilwain
d6e6e8a49c Re-add some accidentally removed license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146694719
2017-02-06 16:45:23 -05:00
mmuller
b70f57b7c7 Update copyright year on all license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146111211
2017-02-02 16:27:22 -05:00
mcilwain
813f837956 Re-add Stackdriver metrics code
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146000275
2017-01-30 15:03:53 -05:00
shikhman
3f9c53b850 Refactor Stackdriver-over-REST specific code to a different package
Downstream users who use gRPC rather than REST don't want to pull down
rest-related dependencies.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145834701
2017-01-30 15:03:53 -05:00
shikhman
e981bad0b3 Rename all encoding methods to start with the word encode for consistency
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143973446
2017-01-12 13:58:31 -05:00
jart
734130aa73 Restructure Maven dependencies in build
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072

We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018

A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55

The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.

Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:

- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
2017-01-09 11:59:04 -05:00
mcilwain
28f6c770c8 Add MOE equivalence for sync on 2016-12-19
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142449539
2017-01-09 11:59:04 -05:00
mcilwain
2b7d580bb3 Run buildifier on codebase to format BUILD files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140362453
2016-11-28 18:15:21 -05:00
jianglai
59d998954c Use correct <a> tag syntax in javadoc @see tag
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137946021
2016-11-02 15:19:34 -04:00
jart
2e81de9954 Make essential Bazel packages publicly visible
This allows separate Bazel projects to reference Nomulus as an external
repository. They can then copy the []
directory structure into their own project and customize the Action
and Module lists for the GAE modules in their own deployment.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136863886
2016-10-24 11:57:00 -04:00
jianglai
cbe6d2b520 Resolve some deprecation warnings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136399461
2016-10-18 10:32:47 -04:00
ctingue
67da55536f Clean up misc Javadoc and deprecation warnings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136156922
2016-10-14 17:00:33 -04:00
shikhman
f76bc70f91 Preserve test logs and test summary output for Kokoro CI runs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135494972
2016-10-14 16:57:43 -04:00
mcilwain
b65b855067 Always use the constructor to make Immutable Collection Builders
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135359669
2016-10-07 14:37:02 -04:00
shikhman
e19546ffb4 Fix encoding error for Stackdriver metrics of cumulative points
The Stackdriver API requires that the end time always be greater than the start
time for cumulative metric points.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134396192
2016-10-03 15:43:16 -04:00
shikhman
4652688585 Add default DistributionFitter implementation to EventMetric
This DistributionFitter is suitable for tracking the latency of network calls.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132400538
2016-09-07 12:20:46 -04:00
shikhman
e5a0392994 Make StackdriverWriter#flush() return early when attempting to flush empty
timeseries

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132359115
2016-09-07 12:10:49 -04:00
shikhman
0e8d9e3859 Force endTimestamp to be strictly greater than or equal to startTimestamp
This fixes an error that we've been seeing from Stackdriver.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132358790
2016-09-07 12:09:23 -04:00
shikhman
dbb9ef80c5 Refactor ConcurrentHashMap creation code to a helper method
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132352263
2016-09-07 12:06:26 -04:00
shikhman
0538c1e210 Add method to create and register EventMetrics
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132346375
2016-09-07 12:03:28 -04:00
shikhman
2ba9b01a13 Add the EventMetric metric type
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132345599
2016-09-07 12:01:58 -04:00
shikhman
969d9483ae Add encoding for the Stackdriver Distribution value type to StackdriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132327732
2016-09-07 11:58:58 -04:00
shikhman
93bf68039b Add lock.unlock() to finally clause in Counter
It is good practice to unlock a lock in a finally clause.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132302413
2016-09-07 11:56:00 -04:00
shikhman
dcb189943b Add the Distribution data type for instrumentation
This is one of a series of CLs adding a new metric type, EventMetric, which
is used for tracking numerical distributions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132103552
2016-09-07 11:54:26 -04:00
shikhman
c11ac3129f Factor out labelValue length check to abstract base class and small name fix
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131902964
2016-09-02 13:47:21 -04:00
shikhman
d77dced024 Fix NPE in StackdriverWriter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131603566
2016-08-30 14:13:50 -04:00
mcilwain
9bdb2ef1f3 Make Stackdriver tests conform to testing practices in rest of codebase
The major changes are using the ExpectedException rule instead of a try/catch
pattern, asserting the message for thrown exceptions, defaulting to JUnit4
unless Mockito is really necessary, and making each test examine one behavior.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131211346
2016-08-26 09:45:10 -04:00
shikhman
b04e54994e Fix open source build
Add missing dependencies to open source build.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130978680
2016-08-22 17:37:01 -04:00
shikhman
91f8b6da38 Change to metrics to keep track of when the metric value was first set
This CL also adds IncrementableMetric#reset() methods to allow resetting the
value and start timestamp of IncrementableMetrics.

This is necessary because some backends, like Stackdriver, use non-monotonic
changes in cumulative metric values to detect timeseries restarts. Tracking and
re-setting the start timestamp allows users to track mostly monotonic metrics
which may have non-monotonic discontinuities.

See https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TimeSeries#Point for
more details.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130795229
2016-08-22 14:44:28 -04:00
shikhman
b6eaba08eb Improve service handling and add unit tests for threading behavior in MetricExporter
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130790170
2016-08-22 14:43:05 -04:00
shikhman
f10a7d8fb0 Enforce monotonicity for IncrementableMetrics
This change enforces that IncrementableMetrics should only monotonically
increase in value, and adds a new increment() method to increment by one, which is slightly faster than incrementBy (due to a lack of non-negative checking) in the common case that the counter should only be incremented by one.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130578421
2016-08-22 14:20:33 -04:00
shikhman
01e38790fd Fix visibility on MetricsReporter#startMetricExporter()
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130576851
2016-08-22 14:19:06 -04:00
shikhman
7a041d066f Improve StackdriverWriter code
Add additional logic to handle cases when a MetricDescriptor is already defined
on the server, fix an NPE in related code, and add additional tests to ensure
that TimeSeries created from native MetricPoints are well-formed.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130555708
2016-08-22 14:17:38 -04:00
shikhman
564bbdef83 Enforce very basic URL-like metric names
This change enforces URL-like metric names to match the behavior of
Stackdriver. The StackdriverWriter no longer prepends a slash, which was a
crutch.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130537347
2016-08-22 14:11:58 -04:00
shikhman
b45c2ca2ee Wire in MetricReporter into the backend servlet to publish metrics
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130468842
2016-08-22 14:07:45 -04:00
shikhman
9b4ef23d92 Change project to projectResource to better match Stackdriver sample code
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130432024
2016-08-22 14:01:50 -04:00
shikhman
64abebec82 Add StackDriver implementation, in monitoring/metrics package
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130287319
2016-08-15 17:12:35 -04:00