zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-05-15 08:57:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

Fetch data from Cloud DNS in parallel

Browse source 
Before pushing an update to Cloud DNS, the CloudDnsWriter needs to read all the domain RRSs from Cloud DNS one by one to know what to delete.

Doing so sequentially results in update times that are too long (approx 200ms per domain, which is 20 seconds per batch of 100) severely limiting our QPS.

This CL uses Concurrent threading to do the Cloud DNS queries in parallel. Unfortunately, my preferred method (Set.parallelStream) doesn't work on App Engine :(

This reduces the per-item time from 200ms to 80ms, which can be further reduced to 50ms if we remove the rate limiter (currently set to 20 per second).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178126877
This commit is contained in:
guyben 2017-12-06 11:30:22 -08:00 • committed by jianglai
parent 735112def6
commit d87f01e7bf
4 changed files with 134 additions and 73 deletions
Download patch file Download diff file Expand all files Collapse all files

20
java/google/registry/util/Concurrent.java
View file

@ -45,12 +45,15 @@ public final class Concurrent {
* @see #transform(Collection, int, Function)
*/
public static <A, B> ImmutableList<B> transform(Collection<A> items, final Function<A, B> funk) {
return transform(items, max(1, min(items.size(), MAX_THREADS)), funk);
return transform(items, MAX_THREADS, funk);
}
/**
* Processes {@code items} in parallel using {@code funk}, with the specified number of threads.
*
* <p>If the maxThreadCount or the number of items is less than 2, will use a non-concurrent
* transform.
*
* <p><b>Note:</b> Spawned threads will inherit the same namespace.
*
* @throws UncheckedExecutionException to wrap the exception thrown by {@code funk}. This will
@ -59,17 +62,18 @@ public final class Concurrent {
*/
public static <A, B> ImmutableList<B> transform(
Collection<A> items,
int threadCount,
int maxThreadCount,
final Function<A, B> funk) {
checkNotNull(funk);
checkNotNull(items);
ThreadFactory threadFactory = currentRequestThreadFactory();
int threadCount = max(1, min(items.size(), maxThreadCount));
ThreadFactory threadFactory = threadCount > 1 ? currentRequestThreadFactory() : null;
if (threadFactory == null) {
// Fall back to non-concurrent transform if we can't get an App Engine thread factory (most
// likely caused by hitting this code from a command-line tool). Default Java system threads
// are not compatible with code that needs to interact with App Engine (such as Objectify),
// which we often have in funk when calling Concurrent.transform().
// For more info see: http://stackoverflow.com/questions/15976406
// Fall back to non-concurrent transform if we only want 1 thread, or if we can't get an App
// Engine thread factory (most likely caused by hitting this code from a command-line tool).
// Default Java system threads are not compatible with code that needs to interact with App
// Engine (such as Objectify), which we often have in funk when calling
// Concurrent.transform(). For more info see: http://stackoverflow.com/questions/15976406
return items.stream().map(funk).collect(toImmutableList());
}
ExecutorService executor = newFixedThreadPool(threadCount, threadFactory);