The probers make a constant stream of create and delete calls, which we don't
want to account for when constructing transaction reports. This change will
cause only real TLDs to log create and delete transaction records.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166737801
This is the last of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
The transfer process is as follows:
GAINING sends a TransferRequest to LOSING
LOSING either acks (TransferApprove), nacks (TransferReject) or does nothing
(auto approve). For acks and autoapproves, we produce a +1 counter for GAINING
and LOSING for domain-gaining/losing-successful for each registrar, to be
reported on the approve date + the transfer grace period. For nacks, we produce
a +1 counter for domain-gaining/losing-nacked for each registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166535579
This is the third of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
We add a +1 counter for either grace or nograce deletes, based on the grace period status of the domain. We then search back in time for DOMAIN_CREATE, DOMAIN_RENEW and DOMAIN_AUTORENEW HistoryEntries off the same resource that happened in their corresponding grace periods (5, 5 and 45 days respectively). All transaction records for these events are then given -1 counters to properly account for cancellations in the NET_CREATE and NET_RENEW fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166506010
Thanks to [] shared libraries at Google now produce valid JSON which allows using JSON.parse. It is safer and faster than goog.json.parse which uses eval by default.
NOTE: All shared libraries producing JSON at Google were changed to produce valid JSON. However, if your code uses a custom way of producing JSON (not using the shared libraries) or if your code parses JSON generated a long time ago and stored, this CL might break you so please review with care.
Design doc: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166454709
To log autorenews, we currently run a mapreduce daily that creates synthetic
billing events for each recurring event past its due time. These are all
parented under the original recurring event, which allows these synthetic events to incorrectly stack on the original mutating entry.
We now explicitly create a new HistoryEntry of type DOMAIN_AUTORENEW to log
autorenews alongside other mutating EPP flows. These also parent DomainTransactionRecords for the NET_RENEWS_1_YEAR field, with the reporting time equal to the billing time (which accounts for the autorenew grace period).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166379700
After working further with domain deletes, I realized we'll need to record multiple reportingTimes under a single historyEntry when issuing a -1 counter to cancel grace-period adds. Since the TLD would be the only shared component within a record, we'll just duplicate it across all records to save an unnecessary layer of hierarchy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166261413
It appears that there are some possible flows where the reducer runs twice on
the same key. Either because of some error in a subsequent key that makes the
entire shard become ignored and retried, or possible some obscure error outside
of the transaction on that key.
The result however is that sometimes the reducer runs on a key that has already
been deleted. We need to check for that to prevent a null pointer exception.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166112173
In standard DNS format, the first thing on an A, NS or DS definition line is a domain label relative to the zone, which in our case is a TLD. However, the generate_zone_files command prints out fully qualified host and domain names, resulting in a discrepancy when compared to the contents of the DNS subsystem. This CL removes the TLD suffix, which should remove one preprocessing step before file comparison.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166103705
Sometimes rdeStaging reduce shards die after the lock is acquired. When that happens - the (automatic) rerun of the shard fails because the lock is in place causing that specific TLD to not stage and await the next call to rdeStaging.
rdeStaging runs every 4 hours, but the current lock lives for 5 hours.
This means that on the next rerun of rdeStaging, the lock still hasn't timed out so it fails again, and we have to wait for the subsequent run - a total delay of 8 hours.
Shortening the lock timeout to be less than the 4 hours rdeStaging rerun time solves this issue.
NOTE: This is just a "quick patch" solution. To really fix the rdeStaging failure we need to fix the lock itself.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166102387
This is the second of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
Adds and renews each result in a +1 counter for the NET_ADDS/RENEWS_#_YR field,
which I've added simple (# of years, add or renew) -> Enum functions to get.
Allocates are just a special case of adds, and are counted in a similar manner.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165963249
Previously, GenerateZoneFilesAction mapreduced its way through all domains and hosts for the specified TLD(s), emitting information for each matching domain and host (subject to constraints like not being deleted and so on). This resulted in host information (aka glue records) for all hosts subordinate to domains in the specified TLD(s). This is incorrect. DNS glue records should only be present for hosts which act as nameservers for their superordinate domains.
The new version of the mapreduce iterates only over domains. When a matching domain is found, a check is made to see whether any subordinate hosts are also nameservers for the domain, in which case host information is generated.
The test was updated to reflect the new reality, and check for a couple additional nuances.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165766472
Right now - if there's an error during DnsWriter.publish*, all the publish from
before that error will be committed, while all the publish after that error
will not.
More than that - in some writers partial publishes can be committed, depending
on implementation.
This defines a new contract that publish* are only committed when .commit is
called. That way any error will simply mean no publish is committed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165708063
This makes it possible to use YamlUtils to configure the proxy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165639295
This is the first of many cls adding explicit logging in all our domain mutation flows to facilitate transaction reporting.
Restores are relatively simple- it happens immediately, so the reporting time is just the time of the HistoryEntry, and we add a single "RESTORED_DOMAINS" count of 1.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165639084
This change adds the persisted data model necessary to facilitate transaction
reporting. TransactionRecord is an embedded repeated class within HistoryEntry
which is only added to when a HistoryEntry is created that counts towards
transaction reporting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165619552
This implements a two-part deletion process for prober domains that were
not deleted properly by the prober (which is usually caused by a transient
network failure). The first time the mapreduce is run, such domains are
soft-deleted, so that their DNS entries can be removed correctly, and then
they are hard-deleted in the subsequent run.
Currently, all domains are hard-deleted the first time this mapreduce
runs, even if they were never soft-deleted correctly, which means that
their published DNS entries won't be correctly handled.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165612973
This is a first step towards porting our database backup comparison tool
(which was written in Crack and subsequently in Python) to Java so that we can
run it in a reasonable amount of time. Ideally, this functionality should be
available in Java, however the only instance of it that I can find is
currently deprecated without a replacement.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165433766
We only log FLOW-LOG-SIGNATURE-METADATA from one place- FlowRunner. As a
result, we can swap the generalized regex for a prefix-only regex, saving a
<strong>lot</strong> of processing for our epp query (which is the most
expensive of the bunch).
I've also changed the test dates from 2017-05 to 2017-06, allowing us to copy-paste
the test data into Bigquery to verify their function. The reason for 2017-06 in particular is because June was the first month that populated all the metadata necessary to generate these reports.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165391715
Currently we exact the client certificate hash from header X-GFE-SSL-Certificate. This works because the proxy running on [] sends the request directly to the AFE via HttpOverRpc, bypassing the frontline GFE, which would strip away this header.
[]
After the proxy moves to GCP we can no longer use that header. Instead we should use X-SSL-Certificate, which does not get stripped by the GFE. In fact the open source build should never have contained X-GFE-SSL-Certificate because obviously external nomulus users have to go through the GFE to reach the registry app and that header would never have survived.
Without changing how the [] proxy works, this CL makes the registry first try to extract the hash from X-GFE-SSL-Certificate, and fallback to X-SSL-Certificate if necessary. This allows the current setup to continue to work, while the new proxy is being tested.
This should not open us up to attacks because even if an attacker uses a proxy that uses X-SSL-Certificate, it still needs to pass OAuth in order to talk to /_dr/epp.
Similarly, we use X-Requested-Servername-SNI as fallback to X-GFE-Requested-Servername-SNI. This can be eliminated altogether when the [] proxy is retired, because the only reason we check if the client request is SNI enabled (by checking the existence of that header) is because the GFE only requests client certificate when SNI is enabled. The GCP proxy does not have that limitation, and also will be only serving one certificate with all SAN listed in it.
Some formatting change is also introduced by the formatter. They seem to be better conforming to the style guide, so I left them there.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165378083
When trying to run the MapReduce for DeleteOldCommitLogsAction, we run into a
lot of DatastoreTimeoutException during CommitLogManifestReader.next.
This causes the entire shard to fail. Since we have a lot of keys (tens of
millions), this is almost guaranteed to happen, dooming the entire MapReduce.
Here is an attempt to recover from the Timeout Exception by saving the state
before the read, then on failure restoring that state and trying again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165172222
Log tasks and task count on the input side of the queue so we can track which
things go in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165026523
This adds Bigquery API client code to generate the activity reports from our
now standardSQL queries. The naming mirrors that of RDE (Staging generates the
reports and uploads them to GCS).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164656344
Attending to this old bug will improve our ability to perform zone comparisons between Datastore and the DNS provider. Right now, zone comparison finds some bogus differences, because the TTL we send to the DNS subsystem doesn't match the TTL we use when generating our local dump files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164635557
Added validation on domain creation, preventing a domain from being created if
it equals an existing TLD. Added domain create tests for domains using
multi-part TLDs that shared suffixes and prefixes. Added host create tests for
hosts using multi-part TLDs that shared suffixes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164297749
This allows admins to access the registrar console without needing to be added
as a registrar contact. If they are a registrar contact, then that registrar
takes precedence.
Tested=In Alpha
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164182623
Attempting to run DeleteOldCommitLogs in prod resulted in a lot of DatastoreTimeoutException errors. The assumption is that attempting to load so many CommitLogManifests (over 200 million of them), when each one has a slight possibility of failure, has a very high probability of error.
The shard aborts after 20 of these errors, and by eliminating as many loads as possible and retrying the remaining loads inside a transaction we are effectively eliminating any exceptions "leaking" out to the mapreduce framework, which will hopefully keep us bellow 20. At least, that's our best guess currently as to why the mapreduce fails.
EppResources are loaded in the map stage to get the revisions, and CommitLogManifests are only loaded in the reduce stage for sanity check so we don't accidentally delete resources we need in prod. Both of these are wrapped in transactNew to make sure they retry individually.
The only "load" not done inside a transaction is the EppResourceIndex, but there's no getting around that without rewriting the EppResourceInputs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164176764
The auth logging has been useful, but it now generates a sizeable percentage of all logging, because it spits out three to five lines for every request in the system. This CL reduces that to two to three. We may eventually want to reduce it further, but this is a good start.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164146182
It was not a problem after all to handle multiple scopes. Also added a temp variable to avoid making the same array conversion over and over.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164002903
When the registrar console code determines that a user has not logged in, it redirects to a login page. But when authenticating as an internal request (which should never happen), the redirection code encountered an exception, resulting in a 500 error.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163867018
This completes the data/functionality migration for multiple DNS writers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163835077
A NullPointerException reported via StackDriver appears to stem from trying to load the claims list right at the moment it was being updated. Since the update only happens once every 12 hours, retrying the load once should fix the problem, if this is really the cause.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163732624
It was buggy (didn't work) and was never actually used.
Why never actually used: for it to be used executeWithLock has to be called
with different requesters on the same lockId. That never happend in the code.
How it was buggy: Logically, the queue is deleted on release of the lock (meaning it was
meaningless the only time it mattered - when the lock isn't taken). In
addition, a different bug meant that having items in the queue prevented the
lock from being released forcing all other tasks to have to wait for lock
timeout even if the task that acquired the lock is long done.
Alternative: fix the queue. This would mean we don't want to delete the lock on release (since we want to keep the queue). Instead, we resave the same lock with expiration date being START_OF_TIME. In addition - we need to fix the .equals used to determine if the lock the same as the acquired lock - instead use some isSame function that ignores the queue.
Note: the queue is dangerous! An item (calling class / action) in the first place of a queue means no other calling class can get that lock. Everything is waiting for the first calling class to be re-run - but that might take a long time (depending on that action's rerun policy) and even might never happen (if for some reason that action decided it was no longer needed without acquiring the lock) - causing all other actions to stall forever!
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163705463
This is a quick fix we can hopefully get out fast before fixing the underlying problem.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163485468
This will make DNS issues easier to debug retroactively as we will be
able to determine, by looking at the logs, if the queue size was growing
unbounded.
Also adds some logging helpers to allow programmatically choosing the level
of logging.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163123783
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).
This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
This also brings the SQL template parameters in-line with the anticipated Bigquery dataset format, and switches from DateTime to the more appropriate LocalDate (since we only need monthly granularity).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162747692
Note that even though the nomulus command line tool now supports multiple
DNS writers for all subcommands, this still won't work quite yet because
the DNS task queue format migration from [] is still in progress.
After next week's push that migration will be complete and we can remove
the final restriction against only having one DNS writer per TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162490399
It should not be multiline, as registrar client ids are single short-ish identifiers with no spaces allowed. There's no way for them to span multiple lines.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162389442
I know the query that finds commit logs already should ignore commit logs
that are too young, but this adds an explicit sanity check for safety's
sake, so we don't have to depend solely on an indexed query for safety.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162386413
