nickfelt 978fc665ea Improve accuracy of ICANN activity EPP/SRS query ... This makes a bunch of accuracy improvements to the ICANN activity EPP/SRS query that parses XML logs. We're going to be getting rid of this code imminently in favor of the new FlowReporter JSON log line which is much easier to interpret correctly, and in fact all of these issues were detected by comparison with that query. Fixing these issues brings this query almost (but not quite) up to par with the new version. The point of checking this in is so that we have evidence that the new version only diverges from the existing version in ways that are desirable and documented herein. Here are the accuracy issues fixed in this CL: 1) false negative - the old query failed to parse the XML entirely for logs with no client ID (e.g. a registrar who isn't logged in attempting to do a domain check), due to an overly restrictive regex 2) false negative - the old query failed to extract the TLD from EPP requests for just ".tld" (no SLD label at all), which would be an invalid request anyway but should probably still be counted in the reporting for that TLD (which the new query does), due to an overly restrictive regex 3) false negative - the old query failed to normalize uppercase TLDs from the raw XML to lowercase, meaning they wouldn't have matched later on in the pipeline 4) false positive - the old query counted dry-runs from the tool as valid EPP requests, the new logging ignores them 5) false positive - if the old query can't extract the TLD (for example, the domain name provided is just "how" instead of a real SLD), it reports a NULL tld, but then the way the overall activity report query works, it considers that metric to apply to *all possible TLDs* (this is necessary behavior for cases where e.g. for contact/host metrics, we do want a null TLD to signify "applies to all TLDs"). In the case of e.g. a domain check though, this results in that domain check being counted for all TLDs' activity reports, even though it should not be counted for any of them. The fix is to manually filter out results with for 'srs-dom-*' metrics with a NULL tld. 6) false negative - old query wasn't counting /check (CheckApiAction) queries, new query does (and for some reason, we have a strange number of these coming in for TLDs other than .how/.soy, pretty much all from AWS IP addresses and with a UserAgent corresponding to the Go HTTP client library) Finally, this also adds more progress printing lines to icann_reporting.py. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=159608369		2017-06-21 10:05:41 -04:00
..
google/registry/reporting	Improve accuracy of ICANN activity EPP/SRS query	2017-06-21 10:05:41 -04:00
BUILD	Run buildifier on codebase to format BUILD files	2016-11-28 18:15:21 -05:00