"keepTasks" is a flag that prevents ReadDnsQueueAction from removing dns-update
tasks from the dns-pull queue, while still launching PublishDnsUpdates tasks to
update the DNS (meaning these tasks will be updated again in the next
ReadDnsQueueAction).
I'm not sure what's the purpose of this flag, but given we now allow multiple
writers (meaning we can already publish the same DNS multiple times) and given
that we can now recover from a bad writer (if a writer doesn't belong to a TLD,
we put the dns-updates queued for that writer back into the dns-pull queue) - I
suspect we don't need it anymore.
Alternative considered: changing this to a "dryRun" flag that won't actually
launch PublishDnsUpdates tasks, but will log which tasks it would have
launched. Decided against it because we will still need to "own" any task for a
significant amount of time if there are many (tens of thousands) tasks in the
queue. Hence a "dryRun" will still affect any actual runs for some time.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=183997187
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
This allows grouping metrics based on the DnsWriter. We can already group by
the TLD, but since a TLD can have multiple writers, and since different writers
perform very differently from one another, it could be important to group by
writer as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182255398
This is a temporary change used for b/71607306 only, and has TODOs to revert at
every change once the bug is done.
We want to check how Cloud DNS handles large (1M+ domain) zones, especially
during resigning. However, due to a separate bug (b/70980350, and maybe another
one) we can't currently create such a large zone in nomulus within the required
4 day timeframe. The most we managed is 300k domains.
We could wait until the bug is fixed, but if there's a problem with Cloud DNS -
we want to find out as fast as possible. Hence, this CL that allows us to
register 1M domains by creating "just" 100k domains in nomulus.
The CL creates a new "MultiplyingCloudDnsWriter" writer, that when publishing a
domain, pretends that we are publishing 10 domains (with 9 additional
"fictional" domains, that get their Datastore data from the actual domain).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180962149
This is done first and formost to stop "empty" commits that cause errors in
publishDnsUpdates. The reason being that the Cloud DNS api fails when there are
no updates at all in a change.
Allowing this is a requirement for the writer to be idempotent - if we delete a
domain, then run the writer to delete it again - we'll get 0 additions and 0
deletions which fails.
This isn't theoretical either - we've seen it happen, causing a
publishDnsUpdates to fail over and over again.
While fixing this, we also remove all RRS that are common between additions and
deletions. This is just an optimization and shouldn't affect behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179525218
Before pushing an update to Cloud DNS, the CloudDnsWriter needs to read all the domain RRSs from Cloud DNS one by one to know what to delete.
Doing so sequentially results in update times that are too long (approx 200ms per domain, which is 20 seconds per batch of 100) severely limiting our QPS.
This CL uses Concurrent threading to do the Cloud DNS queries in parallel. Unfortunately, my preferred method (Set.parallelStream) doesn't work on App Engine :(
This reduces the per-item time from 200ms to 80ms, which can be further reduced to 50ms if we remove the rate limiter (currently set to 20 per second).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178126877
Currently, if for some reason publishDnsUpdates gets a request to publish
domains to a DnsWriter that doesn't belong to said domain - it logs a warning
but published anyway.
This can happen when Writers are changed (swapped for a different writer)
leaving update commands "stuck" with the wrong writer.
Normally you'd expect these update commands to just publish their data and be
on their way. However, if the update fails for some reason (likely - if the
Writer change happened BECAUSE the updates are failing) then the same
publishDnsUpdate command will continue to run forever.
This CL changes the behavior for "publish to wrong DnsWriter" to instead
requeue the batched domains / hosts back to the Dns-pull queue, allowing them
to be re-batched (and hence published) with the correct DnsWriter(s). This
re-batching will take place in ReadDnsQueueAction.java
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177863076
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
Adding the following metrics:
- how long does an update take, per TLD
- number of domains published, per TLD
- number of hosts published, per TLD
All are distributions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172933834
Convert periods to hyphens in multi-part TLDs when using them as a zone name
(cloud-dns doesn't allow periods in zone names).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172007089
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Add cloudDns.{rootUrl, servicePath} to allow us to point an environment at the
Cloud DNS staging API for testing. Make sandbox and alpha point to staging.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170340859
The existing CloudDnsWriter code uses ImmutableMap.Builder to construct the
map of DNS records to update. This has been seen to fail on alpha, presumably
in a cases where host records and domain records produce duplicate updates for
a host.
Convert the Builder to a HashMap, allowing us to safely overwrite existing
records in the case of duplicates.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170103421
This still retains the DnsWriter interface itself for better integration
with Dagger and to preserve the option of having a DNS writer that does
not have this requirement (e.g. because it is idempotent).
This also makes the commit check thread-safe, which is a nice-to-have.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168451114
Right now - if there's an error during DnsWriter.publish*, all the publish from
before that error will be committed, while all the publish after that error
will not.
More than that - in some writers partial publishes can be committed, depending
on implementation.
This defines a new contract that publish* are only committed when .commit is
called. That way any error will simply mean no publish is committed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165708063
Log tasks and task count on the input side of the queue so we can track which
things go in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165026523
Attending to this old bug will improve our ability to perform zone comparisons between Datastore and the DNS provider. Right now, zone comparison finds some bogus differences, because the TTL we send to the DNS subsystem doesn't match the TTL we use when generating our local dump files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164635557
This completes the data/functionality migration for multiple DNS writers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163835077
It was buggy (didn't work) and was never actually used.
Why never actually used: for it to be used executeWithLock has to be called
with different requesters on the same lockId. That never happend in the code.
How it was buggy: Logically, the queue is deleted on release of the lock (meaning it was
meaningless the only time it mattered - when the lock isn't taken). In
addition, a different bug meant that having items in the queue prevented the
lock from being released forcing all other tasks to have to wait for lock
timeout even if the task that acquired the lock is long done.
Alternative: fix the queue. This would mean we don't want to delete the lock on release (since we want to keep the queue). Instead, we resave the same lock with expiration date being START_OF_TIME. In addition - we need to fix the .equals used to determine if the lock the same as the acquired lock - instead use some isSame function that ignores the queue.
Note: the queue is dangerous! An item (calling class / action) in the first place of a queue means no other calling class can get that lock. Everything is waiting for the first calling class to be re-run - but that might take a long time (depending on that action's rerun policy) and even might never happen (if for some reason that action decided it was no longer needed without acquiring the lock) - causing all other actions to stall forever!
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163705463
This is a quick fix we can hopefully get out fast before fixing the underlying problem.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163485468
This will make DNS issues easier to debug retroactively as we will be
able to determine, by looking at the logs, if the queue size was growing
unbounded.
Also adds some logging helpers to allow programmatically choosing the level
of logging.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163123783
Note that even though the nomulus command line tool now supports multiple
DNS writers for all subcommands, this still won't work quite yet because
the DNS task queue format migration from [] is still in progress.
After next week's push that migration will be complete and we can remove
the final restriction against only having one DNS writer per TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162490399
This is written in such a way that it can safely handle task items in the
old format so long as the DNS writer to use for the given TLD is unambiguous
(which it is for now, until we allow multiple DNS writers to be configured).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162293412
We want to be safer and more explicit about the authentication needed by the many actions that exist.
As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
This is a cleanup in preparation for the next change that does a lot
of work with subordinate hosts, to make it easier to reason about in
complex code.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146689904
This allows us to use util methods from within config, which is a useful thing
to be able to do for, e.g., being able to log errors while loading configuration.
It makes sense that the util package should be at the very base of the
class inheritance hierarchy; config seems logically higher than it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=144324273
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072
We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018
A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55
The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.
Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:
- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
This change is required before the migration to MultimapBuilder can be
completed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141208944
This allows separate Bazel projects to reference Nomulus as an external
repository. They can then copy the []
directory structure into their own project and customize the Action
and Module lists for the GAE modules in their own deployment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136863886
This makes the usage of DnsQueue.create() safer, since we're no longer
forced to hardcode a copy of the @Config("dnsWriteLockTimeout") value
within that method. That value is only needed for leaseTasks(), which
is only called in one place (ReadDnsQueueAction), so we can just pass
it in from that callsite.
Also removes an unused overload of leaseTasks() that allowed specifying
a tag, which is a feature we no longer need.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136162491
