RecordAccumulator builds a set of datastore Entity records from a set of
leveldb logfiles in a directory tree (which is how we receive them for
database backup/restore testing).
This CL also refactors some of the logfile test code out of
LevelDbLogReaderTest so that we can reuse it for building test logs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167313553
ComparableEntity allows us to compare Entity objects by both key and
properties, regardless of the order of those properties in the protobufs.
This is necessary for comparing database backups, where we evaluate the
differences in the set of entities in one backup with those of another.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167288489
Previously, I would cancel all the records associated with HistoryEntry that's
available for cancellation. This could cause unexpected behavior if we
cancelled a historyEntry which itself had cancelled records (in effect we would
negate the negation unintentionally). This is easily remedied by only
cancelling records which want to be cancelled.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167204383
This is the first in a series of CLs containing code from an old CL of Dai's that had never been completed, which compares zone data between Datastore and DNS. I had written a script to do this by calling two nomulus commands, but maybe it can be done directly in Java, which would be convenient.
This CL is just the plumbing to check on the status of a Mapreduce. We will need this to know that we can proceed with the next step of comparing the output to the DNS data.
Cloned from CL 134295050 by 'g4 patch'.
Original change by dxy@dxy:zoneman-reader:1939:citc on 2016/09/26 10:34:22.
Add a command for comparing zone data between DNS and datastore
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167188979
One was missing in the ICANN distribution: TreatyStatue-Agent-Arabic-Revoked.smd. I just deleted that one, because we don't need it.
These new files are only valid until June 2018, because the embedded certificate expires then. At that point, ICANN will presumably release a new set of files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167164915
Lessons from the first full run of the mapreduce.
Fixed the "old manifests still referenced" counter, since some of these
manifests could be new.
Added a debug "manifests incorrectly mapped multiple times" that is expected to never happen, but who knows? (this is a sort of sanity check for the mapreduce)
Added an "Epp resource revisions handled" that counts the total references, allowing for the possibility of the same manifest being referenced from multiple resources - this is expected to be equal to the map's "Epp resource revisions found" counter.
Added a "commit log manifests referenced multiple times" to see how often the same manifest is linked to from different epp resources.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166846879
It turns out the Bigquery JSON api selects its validator exclusively through
the useLegacySql flag (the #standardSQL directive isn't considered). To fix
this, we add back the explicit flag.
This also logs unexpected API errors, instead of allowing the job to quietly fail.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166757569
The probers make a constant stream of create and delete calls, which we don't
want to account for when constructing transaction reports. This change will
cause only real TLDs to log create and delete transaction records.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166737801
This is the last of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
The transfer process is as follows:
GAINING sends a TransferRequest to LOSING
LOSING either acks (TransferApprove), nacks (TransferReject) or does nothing
(auto approve). For acks and autoapproves, we produce a +1 counter for GAINING
and LOSING for domain-gaining/losing-successful for each registrar, to be
reported on the approve date + the transfer grace period. For nacks, we produce
a +1 counter for domain-gaining/losing-nacked for each registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166535579
This is the third of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
We add a +1 counter for either grace or nograce deletes, based on the grace period status of the domain. We then search back in time for DOMAIN_CREATE, DOMAIN_RENEW and DOMAIN_AUTORENEW HistoryEntries off the same resource that happened in their corresponding grace periods (5, 5 and 45 days respectively). All transaction records for these events are then given -1 counters to properly account for cancellations in the NET_CREATE and NET_RENEW fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166506010
Thanks to [] shared libraries at Google now produce valid JSON which allows using JSON.parse. It is safer and faster than goog.json.parse which uses eval by default.
NOTE: All shared libraries producing JSON at Google were changed to produce valid JSON. However, if your code uses a custom way of producing JSON (not using the shared libraries) or if your code parses JSON generated a long time ago and stored, this CL might break you so please review with care.
Design doc: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166454709
To log autorenews, we currently run a mapreduce daily that creates synthetic
billing events for each recurring event past its due time. These are all
parented under the original recurring event, which allows these synthetic events to incorrectly stack on the original mutating entry.
We now explicitly create a new HistoryEntry of type DOMAIN_AUTORENEW to log
autorenews alongside other mutating EPP flows. These also parent DomainTransactionRecords for the NET_RENEWS_1_YEAR field, with the reporting time equal to the billing time (which accounts for the autorenew grace period).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166379700
After working further with domain deletes, I realized we'll need to record multiple reportingTimes under a single historyEntry when issuing a -1 counter to cancel grace-period adds. Since the TLD would be the only shared component within a record, we'll just duplicate it across all records to save an unnecessary layer of hierarchy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166261413
It appears that there are some possible flows where the reducer runs twice on
the same key. Either because of some error in a subsequent key that makes the
entire shard become ignored and retried, or possible some obscure error outside
of the transaction on that key.
The result however is that sometimes the reducer runs on a key that has already
been deleted. We need to check for that to prevent a null pointer exception.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166112173
In standard DNS format, the first thing on an A, NS or DS definition line is a domain label relative to the zone, which in our case is a TLD. However, the generate_zone_files command prints out fully qualified host and domain names, resulting in a discrepancy when compared to the contents of the DNS subsystem. This CL removes the TLD suffix, which should remove one preprocessing step before file comparison.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166103705
Sometimes rdeStaging reduce shards die after the lock is acquired. When that happens - the (automatic) rerun of the shard fails because the lock is in place causing that specific TLD to not stage and await the next call to rdeStaging.
rdeStaging runs every 4 hours, but the current lock lives for 5 hours.
This means that on the next rerun of rdeStaging, the lock still hasn't timed out so it fails again, and we have to wait for the subsequent run - a total delay of 8 hours.
Shortening the lock timeout to be less than the 4 hours rdeStaging rerun time solves this issue.
NOTE: This is just a "quick patch" solution. To really fix the rdeStaging failure we need to fix the lock itself.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166102387
The current code fails if there's no X-* header even if there is a X-GFE-*
header, although just having one of them should be enough.
Bug was introduced in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166091723
This is the second of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
Adds and renews each result in a +1 counter for the NET_ADDS/RENEWS_#_YR field,
which I've added simple (# of years, add or renew) -> Enum functions to get.
Allocates are just a special case of adds, and are counted in a similar manner.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165963249
Previously, GenerateZoneFilesAction mapreduced its way through all domains and hosts for the specified TLD(s), emitting information for each matching domain and host (subject to constraints like not being deleted and so on). This resulted in host information (aka glue records) for all hosts subordinate to domains in the specified TLD(s). This is incorrect. DNS glue records should only be present for hosts which act as nameservers for their superordinate domains.
The new version of the mapreduce iterates only over domains. When a matching domain is found, a check is made to see whether any subordinate hosts are also nameservers for the domain, in which case host information is generated.
The test was updated to reflect the new reality, and check for a couple additional nuances.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165766472
Right now - if there's an error during DnsWriter.publish*, all the publish from
before that error will be committed, while all the publish after that error
will not.
More than that - in some writers partial publishes can be committed, depending
on implementation.
This defines a new contract that publish* are only committed when .commit is
called. That way any error will simply mean no publish is committed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165708063
This makes it possible to use YamlUtils to configure the proxy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165639295
This is the first of many cls adding explicit logging in all our domain mutation flows to facilitate transaction reporting.
Restores are relatively simple- it happens immediately, so the reporting time is just the time of the HistoryEntry, and we add a single "RESTORED_DOMAINS" count of 1.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165639084
This change adds the persisted data model necessary to facilitate transaction
reporting. TransactionRecord is an embedded repeated class within HistoryEntry
which is only added to when a HistoryEntry is created that counts towards
transaction reporting.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165619552
This implements a two-part deletion process for prober domains that were
not deleted properly by the prober (which is usually caused by a transient
network failure). The first time the mapreduce is run, such domains are
soft-deleted, so that their DNS entries can be removed correctly, and then
they are hard-deleted in the subsequent run.
Currently, all domains are hard-deleted the first time this mapreduce
runs, even if they were never soft-deleted correctly, which means that
their published DNS entries won't be correctly handled.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165612973
Second attempt at removing this test by simply removing from the generated
test rules.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165575517
This is a first step towards porting our database backup comparison tool
(which was written in Crack and subsequently in Python) to Java so that we can
run it in a reasonable amount of time. Ideally, this functionality should be
available in Java, however the only instance of it that I can find is
currently deprecated without a replacement.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165433766
We only log FLOW-LOG-SIGNATURE-METADATA from one place- FlowRunner. As a
result, we can swap the generalized regex for a prefix-only regex, saving a
<strong>lot</strong> of processing for our epp query (which is the most
expensive of the bunch).
I've also changed the test dates from 2017-05 to 2017-06, allowing us to copy-paste
the test data into Bigquery to verify their function. The reason for 2017-06 in particular is because June was the first month that populated all the metadata necessary to generate these reports.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165391715
Currently we exact the client certificate hash from header X-GFE-SSL-Certificate. This works because the proxy running on [] sends the request directly to the AFE via HttpOverRpc, bypassing the frontline GFE, which would strip away this header.
[]
After the proxy moves to GCP we can no longer use that header. Instead we should use X-SSL-Certificate, which does not get stripped by the GFE. In fact the open source build should never have contained X-GFE-SSL-Certificate because obviously external nomulus users have to go through the GFE to reach the registry app and that header would never have survived.
Without changing how the [] proxy works, this CL makes the registry first try to extract the hash from X-GFE-SSL-Certificate, and fallback to X-SSL-Certificate if necessary. This allows the current setup to continue to work, while the new proxy is being tested.
This should not open us up to attacks because even if an attacker uses a proxy that uses X-SSL-Certificate, it still needs to pass OAuth in order to talk to /_dr/epp.
Similarly, we use X-Requested-Servername-SNI as fallback to X-GFE-Requested-Servername-SNI. This can be eliminated altogether when the [] proxy is retired, because the only reason we check if the client request is SNI enabled (by checking the existence of that header) is because the GFE only requests client certificate when SNI is enabled. The GCP proxy does not have that limitation, and also will be only serving one certificate with all SAN listed in it.
Some formatting change is also introduced by the formatter. They seem to be better conforming to the style guide, so I left them there.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165378083
*** Reason for rollback ***
Breaks kokoro build, still investigating the fix.
*** Original change description ***
Remove RdeUploadActionTest from FOSS tests
Remove from the FOSS test suite until we can figure out why it's timing out.
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165359201
Remove from the FOSS test suite until we can figure out why it's timing out.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165340539
When trying to run the MapReduce for DeleteOldCommitLogsAction, we run into a
lot of DatastoreTimeoutException during CommitLogManifestReader.next.
This causes the entire shard to fail. Since we have a lot of keys (tens of
millions), this is almost guaranteed to happen, dooming the entire MapReduce.
Here is an attempt to recover from the Timeout Exception by saving the state
before the read, then on failure restoring that state and trying again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165172222
Log tasks and task count on the input side of the queue so we can track which
things go in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165026523
Set SystemProperty.environment to "Development" so we don't try to actually
use GCS.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165019677
Fix a verification call to a method that requires it's arg checked that is
breaking in the FOSS build.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164974785
Fix a few uses of Optional and any() in the IcannReporting test drivers so
that they work on bazel in JDK7.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164888832
This adds Bigquery API client code to generate the activity reports from our
now standardSQL queries. The naming mirrors that of RDE (Staging generates the
reports and uploads them to GCS).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164656344
Attending to this old bug will improve our ability to perform zone comparisons between Datastore and the DNS provider. Right now, zone comparison finds some bogus differences, because the TTL we send to the DNS subsystem doesn't match the TTL we use when generating our local dump files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164635557
Added to make it clear that the order of the keys in the map is important. A
better solution would be, of course, to make the order unimportant.
This can be done by using ImmutableSortedMap either as input or with copyOf
the input. But the create function is only used once, so it doesn't seem worth the effort - as long as we make sure to throw an error if the order is wrong.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164515959
Added validation on domain creation, preventing a domain from being created if
it equals an existing TLD. Added domain create tests for domains using
multi-part TLDs that shared suffixes and prefixes. Added host create tests for
hosts using multi-part TLDs that shared suffixes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164297749
