This is a temporary change used for b/71607306 only, and has TODOs to revert at
every change once the bug is done.
We want to check how Cloud DNS handles large (1M+ domain) zones, especially
during resigning. However, due to a separate bug (b/70980350, and maybe another
one) we can't currently create such a large zone in nomulus within the required
4 day timeframe. The most we managed is 300k domains.
We could wait until the bug is fixed, but if there's a problem with Cloud DNS -
we want to find out as fast as possible. Hence, this CL that allows us to
register 1M domains by creating "just" 100k domains in nomulus.
The CL creates a new "MultiplyingCloudDnsWriter" writer, that when publishing a
domain, pretends that we are publishing 10 domains (with 9 additional
"fictional" domains, that get their Datastore data from the actual domain).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=180962149
This is done first and formost to stop "empty" commits that cause errors in
publishDnsUpdates. The reason being that the Cloud DNS api fails when there are
no updates at all in a change.
Allowing this is a requirement for the writer to be idempotent - if we delete a
domain, then run the writer to delete it again - we'll get 0 additions and 0
deletions which fails.
This isn't theoretical either - we've seen it happen, causing a
publishDnsUpdates to fail over and over again.
While fixing this, we also remove all RRS that are common between additions and
deletions. This is just an optimization and shouldn't affect behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179525218
Before pushing an update to Cloud DNS, the CloudDnsWriter needs to read all the domain RRSs from Cloud DNS one by one to know what to delete.
Doing so sequentially results in update times that are too long (approx 200ms per domain, which is 20 seconds per batch of 100) severely limiting our QPS.
This CL uses Concurrent threading to do the Cloud DNS queries in parallel. Unfortunately, my preferred method (Set.parallelStream) doesn't work on App Engine :(
This reduces the per-item time from 200ms to 80ms, which can be further reduced to 50ms if we remove the rate limiter (currently set to 20 per second).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178126877
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
Convert periods to hyphens in multi-part TLDs when using them as a zone name
(cloud-dns doesn't allow periods in zone names).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172007089
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Add cloudDns.{rootUrl, servicePath} to allow us to point an environment at the
Cloud DNS staging API for testing. Make sandbox and alpha point to staging.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170340859
The existing CloudDnsWriter code uses ImmutableMap.Builder to construct the
map of DNS records to update. This has been seen to fail on alpha, presumably
in a cases where host records and domain records produce duplicate updates for
a host.
Convert the Builder to a HashMap, allowing us to safely overwrite existing
records in the case of duplicates.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170103421
This still retains the DnsWriter interface itself for better integration
with Dagger and to preserve the option of having a DNS writer that does
not have this requirement (e.g. because it is idempotent).
This also makes the commit check thread-safe, which is a nice-to-have.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168451114
Right now - if there's an error during DnsWriter.publish*, all the publish from
before that error will be committed, while all the publish after that error
will not.
More than that - in some writers partial publishes can be committed, depending
on implementation.
This defines a new contract that publish* are only committed when .commit is
called. That way any error will simply mean no publish is committed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165708063
Attending to this old bug will improve our ability to perform zone comparisons between Datastore and the DNS provider. Right now, zone comparison finds some bogus differences, because the TTL we send to the DNS subsystem doesn't match the TTL we use when generating our local dump files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164635557
This allows us to use util methods from within config, which is a useful thing
to be able to do for, e.g., being able to log errors while loading configuration.
It makes sense that the util package should be at the very base of the
class inheritance hierarchy; config seems logically higher than it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=144324273
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
This allows separate Bazel projects to reference Nomulus as an external
repository. They can then copy the []
directory structure into their own project and customize the Action
and Module lists for the GAE modules in their own deployment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136863886
It is replaced by loadByForeignKey(), which does the same thing that
loadByUniqueId() did for contacts, hosts, and domains, and also
loadDomainApplication(), which loads domain application by ROID. This eliminates
the ugly mode-switching of attemping to load by other foreign key or ROID.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133980156
I forgot to inject the CreateTldCommand/UpdateTldCommand commands, which now
need to be injected so that the list of available DnsWriter implementations is
instantiated. This CL also adds a new DI Set<String> with just the name of the
writer, so that the instantiated writer map (which may have many DI dependencies)
doesn't need to be pulled in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=130042215
This CL has minimal[*] immediate impact on the targets, but it has two consequences moving forward:
- When [] moves to Java 8, compatible_with=appengine targets will remain on Java 7. (If this target is a non-java_* target, you can ignore this bullet.)
- compatible_with=appengine targets can depend only on other compatible_with=appengine targets. (Fortunately, most common libraries will be compatible_with=appengine: []
This CL is one of thousands in a migration on the critical path for Java 8, so please respond promptly, and, if you are the wrong reviewer, please reassign the CL.
[*] It does mean that your target will go back to being compiled to Java 7 bytecode after its recent switch to Java 8 ([]If your target compiles the same files as another non-compatible_with=appengine target, this CL may break one_definition_test targets.
More information:
[]
Tested:
[] build --nobuild
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129660908
This is one of several CLs in a sequence for allowing per-TLD DNS
implementations.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129445641
The plumbing to support different DnsWriter implementations per TLD
using DI will come in a later CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129375856
