Improve efficiency of async contact and host deletion with batching

This allows handling of N asynchronous deletion requests simultaneously instead
of just 1.  An accumulation pull queue is used for deletion requests, and the
async deletion [] is now fired off whenever that pull queue isn't empty,
and processes many tasks at once.  This doesn't particularly take more time,
because the bulk of the cost of the async delete operation is simply iterating
over all DomainBases (which has to happen regardless of how many contacts and
hosts are being deleted).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133169336
This commit is contained in:
mcilwain 2016-09-14 13:54:16 -07:00 committed by Ben McIlwain
parent 75d9268ecd
commit 00ea99960a
22 changed files with 1186 additions and 16 deletions

View file

@ -229,6 +229,7 @@ public interface RegistryConfig {
* fails the delete). Without this delay, the mapreduce might have started before the domain flow
* committed, and could potentially miss the reference.
*/
// TODO(b/26140521): Remove this configuration option along with non-batched async operations.
public Duration getAsyncDeleteFlowMapreduceDelay();
/**
@ -237,6 +238,7 @@ public interface RegistryConfig {
* This should be ~orders of magnitude larger than the rate on the queue, in order to prevent
* the logs from filling up with unnecessarily failures.
*/
// TODO(b/26140521): Remove this configuration option along with non-batched async operations.
public Duration getAsyncFlowFailureBackoff();
// XXX: Please consider using ConfigModule instead of adding new methods to this file.