Sometimes requests "die" suddenly, without going through catch/finally blocks.
If this happens, any lock they own will remain locked until it times out (which
can take hours in some cases).
This cl implicitly unlocks any lock if the owner of the lock isn't running
anymore.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168880938
Also adds a "resave all epp" cron job that's needed for the delete to work correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168879965
We had been allowing lookups by nameserver name using a wildcard and suffix if the suffix was a domain name. That's ok if the domain name is one we manage, but doesn't work efficiently otherwise. A lookup of ns*.zombo.com would require us to search for all nameservers beginning with ns (which could be almost all of them), then loop through until we found those ending with .zombo.com. So we are going to prohibit suffixes after the TLD unless the suffix is an in-bailiwick domain.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168835732
We're planning on adding a cronjob to run this mapreduce monthly, so
we may as well also project the resources being re-saved to the present
time so as to handle pending transfers, grace periods, and such. This will
make the BigQuery exports more useful.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168831056
Void tags (e.g. <img>) couldn't have a closing tag (e.g. </img> is invalid). Non-void tags (e.g. <div>) couldn't be self closing (e.g. <div/> is invalid) and must be closed explicitly (e.g. with </div>). This CL fixes the tags which also prepares the templates for stricthtml which enforces it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168829028
The bad prober domain data has since been deleted, so we no longer need
to handle the case where these Keys point to entities that don't exist.
This mostly reverses []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168687701
This pattern will mainly be used for data migrations, i.e. updating all
HistoryEntries' DomainTransactionRecords to the new schema.
TESTED=Ran in alpha, the underlying data dropped non-Objectify fields as
expected.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168684356
Implement toplevel class that reads in two database backups and displays
diffs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168592124
This still retains the DnsWriter interface itself for better integration
with Dagger and to preserve the option of having a DNS writer that does
not have this requirement (e.g. because it is idempotent).
This also makes the commit check thread-safe, which is a nice-to-have.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168451114
RDAP filters some contact data when the request is not authenticated as coming from a user associated with the registrar owning the contact. This CL adds an exception for admin users, which for the App Engine Users API are defined as App Engine project viewers. This means that the registry team will always get all information when logged in. This will also be useful when building tools that use RDAP output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168293820
This migration does not change any behavior.
More information: []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168248439
Allow superusers to change the grace period and allow
superusers to change the pending delete length.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168028545
This CL changes the RDAP responses. When the requester asks for information about a domain, and is not logged in as the owning registrar, no contact information is shown. When the requester asks for information about a contact, and is not logged in as the owner registrar, the existence of the contact is shown, but not any personal data (the existence is shown to make things easier to test).
The login uses the same functionality as the registrar console.
For the most part, this CL does not include the necessary tests to make sure that data is not returned when not logged in. The CL is so large that I didn't want to burden it further. Those tests will be added in a follow-on CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168022034
This is needed for the Lock.java enhancement where a lock will be implicitly
released if the request owning it dies.
No matter which solution we want for refactoring the Lock itself, we will need this class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167600314
This is to fix Kokoro, given the directory added in [] Also, added forgotten keyring/api directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598465
Allow superusers to change the transfer period to zero years and allow
superusers to change the automatic transfer length.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598314
Logging will make it easier to troubleshoot WhoisServer exceptions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167596453
We create an injectable LockHandler that just calls the static
Lock.executeWithLocks function.
I'm not sure what's the correct place to put the LockHandler. I think
model/server is only appropriate for the actual datastore lock. This is a "per request" lock, so maybe request/lock?
-----------------------------
This is the initial step in adding the "lock implicitly released on request death" feature, but it's also useful on its own - easier to test Actions when we can use a fake lock.
To keep this CL simple, we keep using the old Lock as is in most places. We just choose a single example to convert to LockHandler to showcase it. Converting all other uses will be in a subsequent CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167357564
RecordAccumulator builds a set of datastore Entity records from a set of
leveldb logfiles in a directory tree (which is how we receive them for
database backup/restore testing).
This CL also refactors some of the logfile test code out of
LevelDbLogReaderTest so that we can reuse it for building test logs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167313553
ComparableEntity allows us to compare Entity objects by both key and
properties, regardless of the order of those properties in the protobufs.
This is necessary for comparing database backups, where we evaluate the
differences in the set of entities in one backup with those of another.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167288489
Previously, I would cancel all the records associated with HistoryEntry that's
available for cancellation. This could cause unexpected behavior if we
cancelled a historyEntry which itself had cancelled records (in effect we would
negate the negation unintentionally). This is easily remedied by only
cancelling records which want to be cancelled.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167204383
This is the first in a series of CLs containing code from an old CL of Dai's that had never been completed, which compares zone data between Datastore and DNS. I had written a script to do this by calling two nomulus commands, but maybe it can be done directly in Java, which would be convenient.
This CL is just the plumbing to check on the status of a Mapreduce. We will need this to know that we can proceed with the next step of comparing the output to the DNS data.
Cloned from CL 134295050 by 'g4 patch'.
Original change by dxy@dxy:zoneman-reader:1939:citc on 2016/09/26 10:34:22.
Add a command for comparing zone data between DNS and datastore
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167188979
Lessons from the first full run of the mapreduce.
Fixed the "old manifests still referenced" counter, since some of these
manifests could be new.
Added a debug "manifests incorrectly mapped multiple times" that is expected to never happen, but who knows? (this is a sort of sanity check for the mapreduce)
Added an "Epp resource revisions handled" that counts the total references, allowing for the possibility of the same manifest being referenced from multiple resources - this is expected to be equal to the map's "Epp resource revisions found" counter.
Added a "commit log manifests referenced multiple times" to see how often the same manifest is linked to from different epp resources.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166846879
It turns out the Bigquery JSON api selects its validator exclusively through
the useLegacySql flag (the #standardSQL directive isn't considered). To fix
this, we add back the explicit flag.
This also logs unexpected API errors, instead of allowing the job to quietly fail.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166757569
The probers make a constant stream of create and delete calls, which we don't
want to account for when constructing transaction reports. This change will
cause only real TLDs to log create and delete transaction records.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166737801
This is the last of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
The transfer process is as follows:
GAINING sends a TransferRequest to LOSING
LOSING either acks (TransferApprove), nacks (TransferReject) or does nothing
(auto approve). For acks and autoapproves, we produce a +1 counter for GAINING
and LOSING for domain-gaining/losing-successful for each registrar, to be
reported on the approve date + the transfer grace period. For nacks, we produce
a +1 counter for domain-gaining/losing-nacked for each registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166535579
This is the third of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
We add a +1 counter for either grace or nograce deletes, based on the grace period status of the domain. We then search back in time for DOMAIN_CREATE, DOMAIN_RENEW and DOMAIN_AUTORENEW HistoryEntries off the same resource that happened in their corresponding grace periods (5, 5 and 45 days respectively). All transaction records for these events are then given -1 counters to properly account for cancellations in the NET_CREATE and NET_RENEW fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166506010
Thanks to [] shared libraries at Google now produce valid JSON which allows using JSON.parse. It is safer and faster than goog.json.parse which uses eval by default.
NOTE: All shared libraries producing JSON at Google were changed to produce valid JSON. However, if your code uses a custom way of producing JSON (not using the shared libraries) or if your code parses JSON generated a long time ago and stored, this CL might break you so please review with care.
Design doc: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166454709
To log autorenews, we currently run a mapreduce daily that creates synthetic
billing events for each recurring event past its due time. These are all
parented under the original recurring event, which allows these synthetic events to incorrectly stack on the original mutating entry.
We now explicitly create a new HistoryEntry of type DOMAIN_AUTORENEW to log
autorenews alongside other mutating EPP flows. These also parent DomainTransactionRecords for the NET_RENEWS_1_YEAR field, with the reporting time equal to the billing time (which accounts for the autorenew grace period).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166379700
After working further with domain deletes, I realized we'll need to record multiple reportingTimes under a single historyEntry when issuing a -1 counter to cancel grace-period adds. Since the TLD would be the only shared component within a record, we'll just duplicate it across all records to save an unnecessary layer of hierarchy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166261413
It appears that there are some possible flows where the reducer runs twice on
the same key. Either because of some error in a subsequent key that makes the
entire shard become ignored and retried, or possible some obscure error outside
of the transaction on that key.
The result however is that sometimes the reducer runs on a key that has already
been deleted. We need to check for that to prevent a null pointer exception.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166112173
In standard DNS format, the first thing on an A, NS or DS definition line is a domain label relative to the zone, which in our case is a TLD. However, the generate_zone_files command prints out fully qualified host and domain names, resulting in a discrepancy when compared to the contents of the DNS subsystem. This CL removes the TLD suffix, which should remove one preprocessing step before file comparison.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166103705
Sometimes rdeStaging reduce shards die after the lock is acquired. When that happens - the (automatic) rerun of the shard fails because the lock is in place causing that specific TLD to not stage and await the next call to rdeStaging.
rdeStaging runs every 4 hours, but the current lock lives for 5 hours.
This means that on the next rerun of rdeStaging, the lock still hasn't timed out so it fails again, and we have to wait for the subsequent run - a total delay of 8 hours.
Shortening the lock timeout to be less than the 4 hours rdeStaging rerun time solves this issue.
NOTE: This is just a "quick patch" solution. To really fix the rdeStaging failure we need to fix the lock itself.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166102387
This is the second of many cls adding explicit logging in all our domain
mutation flows to facilitate transaction reporting.
Adds and renews each result in a +1 counter for the NET_ADDS/RENEWS_#_YR field,
which I've added simple (# of years, add or renew) -> Enum functions to get.
Allocates are just a special case of adds, and are counted in a similar manner.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165963249
Previously, GenerateZoneFilesAction mapreduced its way through all domains and hosts for the specified TLD(s), emitting information for each matching domain and host (subject to constraints like not being deleted and so on). This resulted in host information (aka glue records) for all hosts subordinate to domains in the specified TLD(s). This is incorrect. DNS glue records should only be present for hosts which act as nameservers for their superordinate domains.
The new version of the mapreduce iterates only over domains. When a matching domain is found, a check is made to see whether any subordinate hosts are also nameservers for the domain, in which case host information is generated.
The test was updated to reflect the new reality, and check for a couple additional nuances.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165766472
Right now - if there's an error during DnsWriter.publish*, all the publish from
before that error will be committed, while all the publish after that error
will not.
More than that - in some writers partial publishes can be committed, depending
on implementation.
This defines a new contract that publish* are only committed when .commit is
called. That way any error will simply mean no publish is committed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165708063
