Commit graph

30 commits

Author SHA1 Message Date
guyben
fa447ce37e Fix soft delete for possible double-map of domain
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167040455
2017-09-12 14:57:34 -04:00
larryruili
7fb44e4f66 Parent recurring billing events under separate HistoryEntries
To log autorenews, we currently run a mapreduce daily that creates synthetic
billing events for each recurring event past its due time. These are all
parented under the original recurring event, which allows these synthetic events to incorrectly stack on the original mutating entry.

We now explicitly create a new HistoryEntry of type DOMAIN_AUTORENEW to log
autorenews alongside other mutating EPP flows. These also parent DomainTransactionRecords for the NET_RENEWS_1_YEAR field, with the reporting time equal to the billing time (which accounts for the autorenew grace period).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166379700
2017-08-29 17:08:27 -04:00
mcilwain
46f175e078 Better handle deletion of prober domains
This implements a two-part deletion process for prober domains that were
not deleted properly by the prober (which is usually caused by a transient
network failure). The first time the mapreduce is run, such domains are
soft-deleted, so that their DNS entries can be removed correctly, and then
they are hard-deleted in the subsequent run.

Currently, all domains are hard-deleted the first time this mapreduce
runs, even if they were never soft-deleted correctly, which means that
their published DNS entries won't be correctly handled.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165612973
2017-08-29 16:32:47 -04:00
mcilwain
bb67841884 Add metrics for async batch operation processing
We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:

* An incrementable metric for each async task processed (split out by
  type of task and result).
* Two event metrics for processing time between when a task is enqueued
  and when it is processed -- tracked separately for contact/host
  deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
  run (this is usually 0). Tracked separately for contact/host deletion
  and DNS refresh on host rename.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
2017-06-05 18:17:09 -04:00
nickfelt
65aaeccfc6 Fix TaskQueueHelper param matching for pull queue tasks
This fixes TaskQueueHelper methods and MatchableTaskInfo so that the .param() matching works for pull queues, by parsing the payload for URL-encoded parameters more liberally.  As such, it updates all the places where formerly we were hacking around this by manually constructing the expected payloads and using TaskMatcher.payload() instead.

It also adds a TaskQueueHelper.assertTasksEnqueued() overload that accepts an Iterable<TaskStateInfo> so that you can cleanly assert that a queue contains the same tasks that were returned via a previous call to getQueueInfo("queue").getTaskInfo().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156604901
2017-05-23 17:22:49 -04:00
nickfelt
40fa9ff022 Fix some direct uses of ForeignKey(C/H/D)Index
Mostly these are calls to ForeignKeyIndex.create() (a static method) via subclasses, which is pretty misleading in this case since the type of the return value has nothing to do with the subclass you're qualifying the static method call with (the returned type depends only on the type of the EppResource parameter).

Note however though that while the style guide indeed prohibits qualifying static member references with things other than the class name, the subclassing case is apparently not considered subject to that prohibition in general:
https://groups.google.com[]d/msg/java-style/8ViX-Rh2_sc/48n2lz5nAAAJ

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154331605
2017-05-03 10:40:21 -04:00
mountford
d35be27b65 Remove some unnecessary uses of MockitoJUnitRunner, which is discouraged
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153589104
2017-04-26 10:42:43 -04:00
larryruili
5047d568de Notify registrars of async contact/host deletions
We now send PendingActionNotificationResponses in our poll messages upon completion of an asynchronous contact or host deletion. This is part 1 of 2, which begins logging Trid in all enqueued Host/Contact deletion flows for use in batch deletions, and optionally consuming the resultant Trid info to emit a Host/ContactPendingActionNotifcationResponse.

Part 2 will make this response emission non-optional, which will happen once the queue is cleared of all non-Trid containing tasks.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=153084197
2017-04-26 10:33:55 -04:00
mcilwain
d2ca4b7234 Refactor main loop of MapreduceEntityCleanupAction
This also tightens up some error-checking conditions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149552065
2017-03-13 10:38:21 -04:00
mountford
5d4287a375 Add mapreduce cleanup action
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149432516
2017-03-13 10:24:57 -04:00
mcilwain
cdadb54acd Refer to Datastore everywhere correctly by its capitalized form
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=147479683
2017-02-17 12:12:12 -05:00
mmuller
b70f57b7c7 Update copyright year on all license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146111211
2017-02-02 16:27:22 -05:00
ctingue
00dab551d7 Fix deleted domain bug in recurring billing expansion
Currently, the recurring billing expansion chokes when looking for existing OneTime events for domains that are deleted or transferred at the MR execution time. To fix this, change the FKI query to look for a key at either execution time or recurrence end time, whichever is earliest.

(This one is distinct from [] in that this bug occurs when domains are [] deleted.)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143693153
2017-01-09 11:59:04 -05:00
ctingue
0cf62be403 Fix recurring expansion when handling transferred or deleted domains
The recurring billing event expansion fails when handling domains that are transferred or deleted before the initial renew. Quietly ignore these recurring events.

(This one is distinct from [] in that this bug occurs when domains currently exist under new ownership.)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143594317
2017-01-09 11:59:04 -05:00
mcilwain
be523f7ce2 Increment the clock in expand recurring billings events test
Also adds some clarity on when to use the non-incrementing version.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143037508
2017-01-09 11:59:04 -05:00
mcilwain
28f6c770c8 Add MOE equivalence for sync on 2016-12-19
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142449539
2017-01-09 11:59:04 -05:00
mountford
0afafeff03 Move VerifyEntityIntegrityAction to batch package
By moving []s into the batch package, which is not included in the frontend service, we pave the way to remove the dependency of frontend on the [] library.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142265351
2016-12-19 11:09:20 -05:00
mountford
669c3dc975 Move the async [] classes out of the flows package
We should be able to remove the dependency on the App Engine [] library from the frontend service, since no []s actually run there. In order to do this, we need to remove the various []-reliant classes from the frontend service build. This CL begins the process by moving the two async "flows" to a different package which is not included in the frontend service.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142159568
2016-12-19 11:09:19 -05:00
ctingue
bf13c2a131 Disable sharding for batch package
The tests in this package are failing for reasons likely related to resource consumption. Sharding an expensive test within a single test runner is probably causing more harm than good, so run this test in a single shard.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141480921
2016-12-13 15:54:47 -05:00
ctingue
e51e220a98 Log the start and finish of tests within ShardableTestCase
Kokoro's logs do not identify which test is allocated to which shard, so explicitly log the test names at start and finish.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140594581
2016-12-06 11:52:46 -05:00
ctingue
6373784858 Revert test size change in ExpandRecurringBillingEventsActionTest
Also adding some temporary logging to the test class in order to diagnose some mysterious behavior on Kokoro. I'll back this out once I get a test run to fail.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140482533
2016-12-06 11:52:46 -05:00
mcilwain
2b7d580bb3 Run buildifier on codebase to format BUILD files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140362453
2016-11-28 18:15:21 -05:00
ctingue
82f6146468 Back out [] make batch tests large
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140379458
2016-11-28 17:40:13 -05:00
ctingue
e0fad37734 Increase timeout on batch package tests
ExpandRecurringBillingEventsActionTest is occasionally timing out on Kokoro

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140363421
2016-11-28 17:37:14 -05:00
ctingue
cfeb1d56ee Increase shards on batch package tests
ExpandRecurringBillingEventsActionTest is occasionally timing out.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140059399
2016-11-28 11:39:27 -05:00
ctingue
35d88a9f8c Rethrow exceptions in ExpandRecurringBillingEventsAction
Also fixes bug in handling of future-dated Recurring billing events.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140048357
2016-11-28 11:34:30 -05:00
ctingue
3c36b6b928 Clean up billing MR work
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136196221
2016-10-14 17:47:46 -04:00
shikhman
f76bc70f91 Preserve test logs and test summary output for Kokoro CI runs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135494972
2016-10-14 16:57:43 -04:00
mcilwain
222ccd912a Centralize creation of MapreduceRunner in unit tests
This also changes the default number of mapper shards in tests to 2, which is
the number of EppResourceIndex buckets in unit tests. Running more shards than
there are buckets causes unnecessary test load.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135520601
2016-10-11 11:27:41 -04:00
mcilwain
b6fbe1f3df Consolidate expand billing events action into 'batch' package
Now it lives alongside the delete prober data action, as well as any future
batch/maintenance tasks that should run periodically.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134435668
2016-10-03 15:49:35 -04:00