I'm making this change ONLY for packages with targets that I'll be removing the appengine constraint from. default_compatible_with is fine in general; I just want to separate the default_compatible_with->compatible_with CLs from the CLs that remove compatible_with.
This is part of the cleanup after []
Tested:
blaze build --nobuild
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159713533
This finally fixes b/37629674 by cutting over the ICANN activity reporting query for EPP/SRS metrics to use the new JSON-based structured log line in FlowReporter, which is much easier to parse and interpret correctly than the old XML logging which was not designed to be ingested in BigQuery.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159633467
This makes a bunch of accuracy improvements to the ICANN activity EPP/SRS query that parses XML logs. We're going to be getting rid of this code imminently in favor of the new FlowReporter JSON log line which is much easier to interpret correctly, and in fact all of these issues were detected by comparison with that query. Fixing these issues brings this query almost (but not quite) up to par with the new version. The point of checking this in is so that we have evidence that the new version only diverges from the existing version in ways that are desirable and documented herein.
Here are the accuracy issues fixed in this CL:
1) false negative - the old query failed to parse the XML entirely for logs with no client ID (e.g. a registrar who isn't logged in attempting to do a domain check), due to an overly restrictive regex
2) false negative - the old query failed to extract the TLD from EPP requests for just ".tld" (no SLD label at all), which would be an invalid request anyway but should probably still be counted in the reporting for that TLD (which the new query does), due to an overly restrictive regex
3) false negative - the old query failed to normalize uppercase TLDs from the raw XML to lowercase, meaning they wouldn't have matched later on in the pipeline
4) false positive - the old query counted dry-runs from the tool as valid EPP requests, the new logging ignores them
5) false positive - if the old query can't extract the TLD (for example, the domain name provided is just "how" instead of a real SLD), it reports a NULL tld, but then the way the overall activity report query works, it considers that metric to apply to *all possible TLDs* (this is necessary behavior for cases where e.g. for contact/host metrics, we do want a null TLD to signify "applies to all TLDs"). In the case of e.g. a domain check though, this results in that domain check being counted for all TLDs' activity reports, even though it should not be counted for any of them. The fix is to manually filter out results with for 'srs-dom-*' metrics with a NULL tld.
6) false negative - old query wasn't counting /check (CheckApiAction) queries, new query does (and for some reason, we have a strange number of these coming in for TLDs other than .how/.soy, pretty much all from AWS IP addresses and with a UserAgent corresponding to the Go HTTP client library)
Finally, this also adds more progress printing lines to icann_reporting.py.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159608369
The affected actions have been changed to check that the user is logged in by [] so this attribute is no longer needed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159572365
private="true" has never done very much unfortunately. There is an alternate
way to mark templates as private by setting 'visibility="private"' which does
get enforced both by the soy compiler and by custom per-language strategies in
the various backends.
Unfortunately, it isn't possible to safely migrate from one to the other since
users may be calling these templates from server side soy which enforces
visibility via a runtime exception. So they are just being deleted.
LSC: https://docs.google.com/document/d/1aOM_tBmanDQolF8mAuB0y29yB76cmYS5qOVYBAzFzyw/edit#
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159565458
We are going to remove the requireLogin attribute from the action attribute, because it is specific to the UserService API. This is used by four actions:
ConsoleUIAction
RegistrarSettingsAction
RegistrarPaymentSetupAction
RegistrarPaymentAction
Instead, these four actions will now check the login status directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159562335
Now that the registration period has been added to DomainApplication, we
can remove this @OnLoad that was populating it for objects that were
missing the period.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159464438
I happened to notice that one domain affected by b/33289763 had existing transfer data, so in that case the data got overwritten to look like a server cancellation (a year after the transfer request, so obviously wrong) rather than creating invalid transfer data, and hence wasn't picked up in the previous fixes. See b/33289763#comment32 for more details.
This CL renames the RemoveDomainTransferDataCommand used to fix the errors to FixDomainTransferDataCommand and changes its behavior to instead reset the TransferData to a server-approval result. It also changes it to load domains by ROID because loadByForeignKey() fails for deleted domains.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159273225
When doing update_registrar, it is now possible to only specify the currencies and the account ids that need updating.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159262119
The --superuser command in the nomulus command-line tool should be
bypassing checks on whether the passed-in registrar client ID has access
to the TLD in question, but currently it is not.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158974462
We currently only have one method for fine level logging. Add another three so that fine level logging can be called just like other levels.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158760803
These shouldn't ever be null, but we have some bad data in production
for prober TLDs left over from the Registry 2.0 transition. Ignoring
null values here is required to finish cleanup for this old data, which
currently cannot even be deleted because it's throwing an NPE when
trying to update these values.
This commit will be reverted after the bad data is cleaned up, likely
sometime next week.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158546840
Now that we've been live with the new code, all of the old tasks
have been processed, and the fallback code to handle the situation
of those fields not being specified is not necessary.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158530336
It turns out that StackdriverWriter uses reflection on the class of
the DistributionFitter instance, so rather than giving it custom
handling for FibonacciFitters, it's easier to turn FibonacciFitter
into a single utility method that takes advantage of existing support
for CustomFitters.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158190147
PDT testing revealed a couple ways in which our WHOIS output was non-compliant. First, the Consistent Labeling & Display policy dictates that the contact IDs must be ROIDs. See rule 11 in https://www.icann.org/resources/pages/rdds-labeling-policy-2017-02-01-en. Second, PDT tests expect that a WHOIS response will treat missing values either by omitting the line entirely, or by including the line with a blank value, but not both. So this is legal:
Phone Number: 123-4567
Phone Number Ext:
Fax Number: 123-4568
Fax Number Ext:
and this is legal:
Phone Number: 123-4567
Fax Number: 123-4568
but this is not:
Phone Number: 123-4567
Phone Number Ext:
Fax Number: 123-4568
In the last example, one extension line is present with a blank value, while the other extension line is omitted. We cannot do both. Therefore, we should update our code to omit lines with no value. Since we can't guarantee that we will always emit all lines that the parse might expect to see, it is safe to use the policy of omitting lines with no value.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158184150
Memcache is already off but now it's not in the code anymore.
This includes removing domain creation failfast, since that is actually
slower now than just running the flow - all you gain is a non-transactional
read over a transactional read, but the cost is that you always pay that
read, which is going to drive up latency.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158183506
I think this comment was meant to be the justification for not using "localhost" (aka a hostname) in the URLs, because jsch would mangle it. However, we already cut over to using "localhost" in [] to avoid a dependency on IPv4, and it's been fine. So this comment no longer makes sense.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158063880
Changed [] to use v1 instead of v1beta1, and replaced v1beta1 with v1 in all the java files.
If there is special build rules for open-source etc. that also need to be updated, or non "TAP-able" tests that need to be run, please check and see if they are OK.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157895888
The command was set up such that an update without any contact types specified would clear out the list, instead of leaving them unchanged, as it should.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157766429
[] opened up the settings on RdeStaging, in order to make it usable by the nomulus tool. But in retrospect, we think that all we needed to do was support the POST method, not loosen the auth settings, since nomulus invokes RdeStaging via a task queue. Removing the looser auth settings will bring this action into line with other backend actions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157262595
This brings the affected actions into line with the settings on other similar actions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157259842
Our build got broken by the deprecation of goog.structs.Map in [] This is a quick fix.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157236744
Convert RestoreCommitLogsCommandTest use of generics and mockito to a form
that works with the kokoro build:
- Replace ImmutableMap<String, Object> with ImmutableMap<String, ?>.
- Replace any() as a matcher for MediaType with an "eq()" matcher.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157148910
We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:
* An incrementable metric for each async task processed (split out by
type of task and result).
* Two event metrics for processing time between when a task is enqueued
and when it is processed -- tracked separately for contact/host
deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
run (this is usually 0). Tracked separately for contact/host deletion
and DNS refresh on host rename.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
A Fibonacci fitter is useful in situations where you want more precision on the
low end than an ExponentialFitter with exponent base 2 provides without the
hassle of dealing with non-integer boundaries, such as would be created by an
exponential fitter with a base of less than 2. Fibonacci fitters are ideal for
integer metrics that are bounded across a certain range, e.g. integers between 1
and 1,000.
This also cleans up some unit test comments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156773367
Move the "restoreCommitLogs" command from the backend module to the tools
module so it's easier to access with nomulus.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156768389
The StatusProhibitsOperationException superclass constructor just wants a raw message to return to the client, so we should pass it an explanation rather than just the superordinate domain name. In fact, I think the superordinate domain name isn't really necessary since it should be obvious from the hostname in the create/update request what the superordinate domain would be.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156747185
It was setting the TLD and watermark date parameters as comma-separated lists. What it really needs to do is set the parameters multiple times, once for each value.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156741051
This fixes TaskQueueHelper methods and MatchableTaskInfo so that the .param() matching works for pull queues, by parsing the payload for URL-encoded parameters more liberally. As such, it updates all the places where formerly we were hacking around this by manually constructing the expected payloads and using TaskMatcher.payload() instead.
It also adds a TaskQueueHelper.assertTasksEnqueued() overload that accepts an Iterable<TaskStateInfo> so that you can cleanly assert that a queue contains the same tasks that were returned via a previous call to getQueueInfo("queue").getTaskInfo().
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156604901
This was needed to correct bad data (LINKED status values on EppResources). The code has been fixed to no longer persist LINKED on any resources and I ran a resave all action yesterday to remove all persisted LINKED status values, so the migration @OnLoad can be safely removed now.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156580334
HttpServlet#destroy() is not called on instance shutdown, so the metrics code was never shut down correctly before.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156360866
WHOIS disclaimer is read from the config yaml file directly and the line breaks
within the text does not contain carriage return, which ICANN requires. This
CL fixes the non-compliance.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156359526
*** Reason for rollback ***
This change is crashing alpha
*** Original change description ***
Change metrics ThreadFactory to produce daemon threads
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156336467
