There are no needs to maintain two versions of the schema gold file now that we do not have any proprietary entities that are not supposed to be exposed in the foss build.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169905583
Based on the original pull request below with some modifications for code drift
over time (including adding handling/testing for the case where superordinate
domains are in pending delete, and thus can't have hosts linked to them).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169698157
This also renames the document to clarify its scope as being all of
Google Cloud Platform, not just App Engine.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169543846
We can easily end up enlisting too many entity groups (separate
DomainApplications) in a TransactionalFlow when loading all applications
tracked by the DomainApplicationIndex. This makes the load operation
transactionless, to avoid overenlisting.
Potential problems:
1. We fail to prevent landrush applications, if a single sunrise application
exists. This is likely fine, except for a brief moment in Sunrush when a
sunrise application is made immediately prior to a landrush application. The
result is we accept an invalid application- which can be mediated manually.
2. We fail to prevent a domain create for a domain with an open application.
This is a little more sinister, but also unlikely unless someone submits an
application immediately before someone tries to create the same domain (sans
application?)
3. We return an invalid DomainCheck response (instead of 'pending allocation').
Not the worst outcome.
4. We reduce the AuctionStatusCommand and GetApplicationIdsCommand to
eventual consistency. Since they're internal tools, that's not too big a deal.
A better solution:
DomainApplications really should just be normalized under a virtualEntityGroup
by fullyQualifiedDomainName, or a hash-bucket like EppResources are. The
DomainApplication -> DomainBase -> EppResource hierarchy seems to be purely for
code reuse, at the cost of Datastore consistency. This would, however, require
quite some refactoring, and a custom resave operation across all
DomainApplications.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169395586
autoescape="strict" is the default so there's no need to require it.
Tested:
TAP found no affected targets
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169387534
This moves us from the oudated google/data XML api to the OnePlatform REST/JSON api, finally silencing the deprecation warnings we've been seeing.
The synchronization algorithm diffs the spreadsheet's current values with its internally sourced values, adding the row to a batch update request if there's a discrepancy. Additional internal data are added as an append operation to the end of the sheet, and any extraneous spreadsheet data is cleared from the spreadsheet.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169273590
LogsExportCursor was only used by ExportLogsTaskServlet, which we removed a long time ago. It's just dead code. The PersistedRangeLong type was only written for use by LogsExportCursor, and since it hasn't picked up new users in 3+ years I don't think we need to keep it around.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169264994
The gcloud datastore cleanup-indexes command takes its input in index.yaml form instead of datastore-indexes.yaml form. This simple translation script allows us to generate one from the other.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169260811
There was a TODO to consider making RDAP queries look up domains by foreign key rather than directly to the domain itself. But since the TLD is a possible search filter, it makes more sense to stick with direct lookup, so we can take advantage of the index on TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169240318
When the domain-contact-postalInfos-footer was self-closed, the resulting contact-postalInfo element improperly included the following hidden input tag. Now that the tag is properly closed, the number of elements in contact-postalInfo should be reduced by 1 (the currently expected numbers are wrong and supposedly were there to have the tests pass).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169096690
When searching for domains by nameserver name or IP address, we fetch the matching nameserver keys, then search for domains by those keys. We limit the number of nameserver keys returned, to avoid arbitrarily large domain queries. This CL adds a warning to the RDAP response if we retrieved the maximum number of nameservers. This may indicate that we have not found all the domains.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168885124
It makes sense for all mapreduces to run in backend, especially onces
that are scheduled regularly to run in cron like this one now. We don't
have many instances configured for the tools service anymore on some
of our environments, so backend is the friendliest place for a mapreduce
to run.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168882122
The lock is used to prevent dual writing of the same output file, and to
prevent us from generating the same data twice (if RdeStaging was run twice).
However, at times when we're creating both RDE and BRDA reports - we use the
same lock for both even though they write to different files and create
different reports.
This causes one of the two to almost always fail (when both are created, which
is once a week I think) delaying that report by several hours.
Note that the "prefix" part of the filenames has the mode in it, so different
modes will not have the same files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168882007
Sometimes requests "die" suddenly, without going through catch/finally blocks.
If this happens, any lock they own will remain locked until it times out (which
can take hours in some cases).
This cl implicitly unlocks any lock if the owner of the lock isn't running
anymore.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168880938
Also adds a "resave all epp" cron job that's needed for the delete to work correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168879965
We had been allowing lookups by nameserver name using a wildcard and suffix if the suffix was a domain name. That's ok if the domain name is one we manage, but doesn't work efficiently otherwise. A lookup of ns*.zombo.com would require us to search for all nameservers beginning with ns (which could be almost all of them), then loop through until we found those ending with .zombo.com. So we are going to prohibit suffixes after the TLD unless the suffix is an in-bailiwick domain.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168835732
We're planning on adding a cronjob to run this mapreduce monthly, so
we may as well also project the resources being re-saved to the present
time so as to handle pending transfers, grace periods, and such. This will
make the BigQuery exports more useful.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168831056
Void tags (e.g. <img>) couldn't have a closing tag (e.g. </img> is invalid). Non-void tags (e.g. <div>) couldn't be self closing (e.g. <div/> is invalid) and must be closed explicitly (e.g. with </div>). This CL fixes the tags which also prepares the templates for stricthtml which enforces it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168829028
The bad prober domain data has since been deleted, so we no longer need
to handle the case where these Keys point to entities that don't exist.
This mostly reverses []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168687701
This pattern will mainly be used for data migrations, i.e. updating all
HistoryEntries' DomainTransactionRecords to the new schema.
TESTED=Ran in alpha, the underlying data dropped non-Objectify fields as
expected.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168684356
Implement toplevel class that reads in two database backups and displays
diffs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168592124
This still retains the DnsWriter interface itself for better integration
with Dagger and to preserve the option of having a DNS writer that does
not have this requirement (e.g. because it is idempotent).
This also makes the commit check thread-safe, which is a nice-to-have.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168451114
RDAP filters some contact data when the request is not authenticated as coming from a user associated with the registrar owning the contact. This CL adds an exception for admin users, which for the App Engine Users API are defined as App Engine project viewers. This means that the registry team will always get all information when logged in. This will also be useful when building tools that use RDAP output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168293820
This migration does not change any behavior.
More information: []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168248439
This CL adds extra tests to make sure that contact data is hidden for RDAP domain and contact requests when not logged in as the proper registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168234733
Allow superusers to change the grace period and allow
superusers to change the pending delete length.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168028545
This CL changes the RDAP responses. When the requester asks for information about a domain, and is not logged in as the owning registrar, no contact information is shown. When the requester asks for information about a contact, and is not logged in as the owner registrar, the existence of the contact is shown, but not any personal data (the existence is shown to make things easier to test).
The login uses the same functionality as the registrar console.
For the most part, this CL does not include the necessary tests to make sure that data is not returned when not logged in. The CL is so large that I didn't want to burden it further. Those tests will be added in a follow-on CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168022034
This is needed for the Lock.java enhancement where a lock will be implicitly
released if the request owning it dies.
No matter which solution we want for refactoring the Lock itself, we will need this class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167600314
This is to fix Kokoro, given the directory added in [] Also, added forgotten keyring/api directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598465
Allow superusers to change the transfer period to zero years and allow
superusers to change the automatic transfer length.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598314
Logging will make it easier to troubleshoot WhoisServer exceptions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167596453
We create an injectable LockHandler that just calls the static
Lock.executeWithLocks function.
I'm not sure what's the correct place to put the LockHandler. I think
model/server is only appropriate for the actual datastore lock. This is a "per request" lock, so maybe request/lock?
-----------------------------
This is the initial step in adding the "lock implicitly released on request death" feature, but it's also useful on its own - easier to test Actions when we can use a fake lock.
To keep this CL simple, we keep using the old Lock as is in most places. We just choose a single example to convert to LockHandler to showcase it. Converting all other uses will be in a subsequent CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167357564
