Specifically, this prevents suspended registrars from creating domains or applications. Pending registrars already can't perform these actions because they get an error message when attempting to log in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170481338
It makes sense for all mapreduces to run in backend, especially onces
that are scheduled regularly to run in cron like this one now. We don't
have many instances configured for the tools service anymore on some
of our environments, so backend is the friendliest place for a mapreduce
to run.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168882122
We're planning on adding a cronjob to run this mapreduce monthly, so
we may as well also project the resources being re-saved to the present
time so as to handle pending transfers, grace periods, and such. This will
make the BigQuery exports more useful.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168831056
Void tags (e.g. <img>) couldn't have a closing tag (e.g. </img> is invalid). Non-void tags (e.g. <div>) couldn't be self closing (e.g. <div/> is invalid) and must be closed explicitly (e.g. with </div>). This CL fixes the tags which also prepares the templates for stricthtml which enforces it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168829028
This pattern will mainly be used for data migrations, i.e. updating all
HistoryEntries' DomainTransactionRecords to the new schema.
TESTED=Ran in alpha, the underlying data dropped non-Objectify fields as
expected.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168684356
Implement toplevel class that reads in two database backups and displays
diffs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168592124
RecordAccumulator builds a set of datastore Entity records from a set of
leveldb logfiles in a directory tree (which is how we receive them for
database backup/restore testing).
This CL also refactors some of the logfile test code out of
LevelDbLogReaderTest so that we can reuse it for building test logs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167313553
ComparableEntity allows us to compare Entity objects by both key and
properties, regardless of the order of those properties in the protobufs.
This is necessary for comparing database backups, where we evaluate the
differences in the set of entities in one backup with those of another.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167288489
This is the first in a series of CLs containing code from an old CL of Dai's that had never been completed, which compares zone data between Datastore and DNS. I had written a script to do this by calling two nomulus commands, but maybe it can be done directly in Java, which would be convenient.
This CL is just the plumbing to check on the status of a Mapreduce. We will need this to know that we can proceed with the next step of comparing the output to the DNS data.
Cloned from CL 134295050 by 'g4 patch'.
Original change by dxy@dxy:zoneman-reader:1939:citc on 2016/09/26 10:34:22.
Add a command for comparing zone data between DNS and datastore
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167188979
To log autorenews, we currently run a mapreduce daily that creates synthetic
billing events for each recurring event past its due time. These are all
parented under the original recurring event, which allows these synthetic events to incorrectly stack on the original mutating entry.
We now explicitly create a new HistoryEntry of type DOMAIN_AUTORENEW to log
autorenews alongside other mutating EPP flows. These also parent DomainTransactionRecords for the NET_RENEWS_1_YEAR field, with the reporting time equal to the billing time (which accounts for the autorenew grace period).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166379700
In standard DNS format, the first thing on an A, NS or DS definition line is a domain label relative to the zone, which in our case is a TLD. However, the generate_zone_files command prints out fully qualified host and domain names, resulting in a discrepancy when compared to the contents of the DNS subsystem. This CL removes the TLD suffix, which should remove one preprocessing step before file comparison.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166103705
Previously, GenerateZoneFilesAction mapreduced its way through all domains and hosts for the specified TLD(s), emitting information for each matching domain and host (subject to constraints like not being deleted and so on). This resulted in host information (aka glue records) for all hosts subordinate to domains in the specified TLD(s). This is incorrect. DNS glue records should only be present for hosts which act as nameservers for their superordinate domains.
The new version of the mapreduce iterates only over domains. When a matching domain is found, a check is made to see whether any subordinate hosts are also nameservers for the domain, in which case host information is generated.
The test was updated to reflect the new reality, and check for a couple additional nuances.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165766472
This is a first step towards porting our database backup comparison tool
(which was written in Crack and subsequently in Python) to Java so that we can
run it in a reasonable amount of time. Ideally, this functionality should be
available in Java, however the only instance of it that I can find is
currently deprecated without a replacement.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=165433766
Attending to this old bug will improve our ability to perform zone comparisons between Datastore and the DNS provider. Right now, zone comparison finds some bogus differences, because the TTL we send to the DNS subsystem doesn't match the TTL we use when generating our local dump files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164635557
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).
This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
Note that even though the nomulus command line tool now supports multiple
DNS writers for all subcommands, this still won't work quite yet because
the DNS task queue format migration from [] is still in progress.
After next week's push that migration will be complete and we can remove
the final restriction against only having one DNS writer per TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162490399
We want to be safer and more explicit about the authentication needed by the many actions that exist.
As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
After this point all data is migrated to use the new canonical
plural version, and subsequent code changes can be made that use
multiple writers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161673486
This makes it take a lot less time to run (roughly a 10X speedup).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161666391
This will allow us to migrate one TLD at a time by refreshing all zones
on the specified TLD after dual-writing is enabled.
Note that the TLDs parameter is required, which seems reasonable given
that almost all imagined use cases would be on a by-TLD basis.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=160294546
I'm moving it out of the scrap folder too because there's nothing else
in there and we do want to retain this indefinitely because it's a useful
tool for performing DNS writer migrations.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=160168902
I happened to notice that one domain affected by b/33289763 had existing transfer data, so in that case the data got overwritten to look like a server cancellation (a year after the transfer request, so obviously wrong) rather than creating invalid transfer data, and hence wasn't picked up in the previous fixes. See b/33289763#comment32 for more details.
This CL renames the RemoveDomainTransferDataCommand used to fix the errors to FixDomainTransferDataCommand and changes its behavior to instead reset the TransferData to a server-approval result. It also changes it to load domains by ROID because loadByForeignKey() fails for deleted domains.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159273225
When doing update_registrar, it is now possible to only specify the currencies and the account ids that need updating.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159262119
Memcache is already off but now it's not in the code anymore.
This includes removing domain creation failfast, since that is actually
slower now than just running the flow - all you gain is a non-transactional
read over a transactional read, but the cost is that you always pay that
read, which is going to drive up latency.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=158183506
The command was set up such that an update without any contact types specified would clear out the list, instead of leaving them unchanged, as it should.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157766429
This brings the affected actions into line with the settings on other similar actions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157259842
It was setting the TLD and watermark date parameters as comma-separated lists. What it really needs to do is set the parameters multiple times, once for each value.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156741051
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.
While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
Now that we've updated the oauth java6 package in the open source build, we
can safely replace "Exception" with "IOException" in the throws clause of the
method that calls AuthorizationCodeInstalledApp.authorize().
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156057190
For the currency specific billing account ids to work properly, any REAL registrar that is allowed to register domains in a certain TLD must have a billing account id for the currency of that TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864711
This helps creating test domains. Also fixed a bunch of bad test cases.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155864065
This tool finds all registrars that have a null name, or a null localizedAddress field, and populates them as follows:
- The registrar's name is updated to its clientId
- The registrar's empty localizedAddress fields are replaced with fake data
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155739946
Moving to KmsKeyring, any command that uses the Keyring must be a
RemoteApiCommand to work.
TESTED: successfully encrypted a file with no logged errors.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155673003
This should be %SOURCE_DATASET% so that it's adjusted appropriately based on the parameter value.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155582718
The absence of these fields causes RDE failures, so they are in effect
required on any functioning registry system. We are currently
experiencing problems in sandbox caused by null values on these fields.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155474895
This is needed to delete an invalid host resulting from a previous bug.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154757425
Currently, if the ID is invalid, parentKey is set to null, causing it to return all history entries. Note that there is still a problem that you cannot look up history entries for entities which have been soft deleted, because the foreign key lookup won't work. That is unfortunate, but at least this simple fix makes things better than they were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154564262
We ran into a bunch of prober deployment issues this past week when
attempting to spin up a new cluster because the newly created prober
TLDs had null values for the dnsWriter field. Given that VoidDnsWriter
exists, we can require that dnsWriter always be set, and have people
use that if DNS publishing is not required.
Also cleans up a bunch of related inconsistent exception messages and
tests not verifying said exception messages properly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=154325830
