This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
Gustavo Lozano confirmed that street addresses should be formatted as a string
if there is only one line, or an nested array of strings if there is more than
one. Luckily we already do it this way, so we just need to update the comment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196845222
Even when the request is not permissioned to see contact information, we should
show information about the owning registrar.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187049833
The RDAP Pilot Program operational profile document indicates that domain
responses should list, in addition to their normal contacts, a special entity
for the registrar.
1.5.12. The domain object in the RDAP response MUST contain an entity with the registrar role (called registrar entity in this section). The handle of the entity MUST be equal to the IANA Registrar ID. A valid fn member MUST be present in the registrar entity. Other members MAY be present in the entity (as specified in RFC6350, the vCard Format Specification and its corresponding JSON mapping RFC7095). Contracted parties MUST include an entity with the abuse role (called Abuse Entity in this section) within the registrar entity. The Abuse Entity MUST include tel and email members, and MAY include other members.
1.5.13. The entity with the registrar role in the RDAP response MUST contain a publicIDs member [RFC7483] to identify the IANA Registrar ID from the IANA’s Registrar ID registry (https://www.iana.org/assignments/registrar-ids/registrar-ids.xhtml). The type value of the publicID object MUST be equal to IANA Registrar ID.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=186797360
Changes the code to be in compliance with the RDAP Pilot Profile document,
which specifies:
1.4.11. If permitted or required by an ICANN agreement provision, waiver, or Consensus Policy, an RDAP response may contain redacted registrant, administrative, technical and/or other contact information. If any information is redacted, the response MUST include a remarks member with title "Data Policy", type "object truncated due to authorization", a description containing the string "Some of the data in this object has been removed" and a links member with the elements rel:alternate and href indicating where the data policy can be found. An entity with redacted information MUST include the "removed" value in the status element.
We were using the "removed" status to indicate deleted contacts and inactive
registrars. Instead, we will now use "inactive", so that we can use "removed"
to indicated redaction.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=185039201
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
This CL also fixes a bug. Registrars were returned in an arbitrary order. This caused cursor-based pagination to fail. Now we always sort by registrar name (even for handle searches), and use the registrar name in the cursor, to ensure proper behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182098187
By default, RDAP entity searches return both contacts and registrars. This CL
adds a new query parameter to request only one or the other. Among other
benefits, this will allow a future CL to permit wildcard searches that return
all registrars.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=181605990
In addition, while adding the tests, I became discontented with the thoroughness of the cursor navigation tests, which checked only the number of items returned, not their proper ordering. So I updated them to be more careful, and backported the changes to the nameserver and entity search tests as well.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179442118
A couple methods were moved to new locations so they are accessible to all types of search queries, not just nameservers like they originally were.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179089014
Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
Also, a couple tests are reworked to make their output more understandable in
case of failure.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
Also, login logic pulled out to helper methods in the test class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.
The default wildcard type is now INVALID rather than NO_WILDCARD.
A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
The code to use this class will come in future CLs, to avoid a huge CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
I am not happy that another index is required, but the Pantheon console shows that domain indexes are much smaller than the other indexes (because there are fewer domains), so it's not adding an appreciable amount of storage space.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173561771
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
We were relying on Dagger to validate the IP address, but that resulted in 500 errors when the IP address was not valid, which is undesirable. Instead, accept the parameters as strings, then convert them to IP addresses and throw a proper error when conversion fails.
Also fixes an improperly specified test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173172516
Usually, the correct order happens automatically, because we are searching on either the key or a specific field like fullyQualifiedDomainName, and the results come back in that order. But when searching by nameserver, where we split the query into multiple "IN" chunks, we have to assemble the result set and order after the fact.
The tests didn't pick up the problem, because the domains and hosts were created in alphabetical order, so it happened to work anyway. The tests have now been changed to create things in reverse order, to test the reordering. Also, the previous arbitrary limit of 1000 nameservers in the intermediate query has been reduced to 300, because we now loop through all nameservers no matter what, rather than stopping when we collect enough domains, so there's more of a penalty for having way too many nameservers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173163121
We no longer find contacts by name if the request is not authorized to see the name.
Several changes cascade from this. Previously, the code assumed that deleted contacts might still have full names, and therefore be searchable. This is not possible in all cases, because Datastore doesn't have the right index to find deleted contacts by name with a matching registrar. However, luckily, this situation can never occur, because contacts always have their name fields nulled out when they are deleted.
So instead, we simply ignore deleted records when searching by name, knowing that none can ever match.
The tests were then changed so that deleted records look the way the really will, meaning devoid of personal information.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172776926
This CL adds the functionality for domain searches. Entities and nameservers have already been handled by previous CLs.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172097922
The nameserver may be external, in which case its TLD will not appear in our
list of valid TLDs, and the search will be rejected erroneously.
Tests for letter case canonicalizations also added at reviewer's suggestion.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171985702
Also sets the charset to UTF8 explicitly, to avoid confusion, and removes apparently unneeded Truth8 include.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171963981
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Also fixed a bug discovered as a result of the more complete testing!
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171156304
This CL adds the functionality for contact and registrar searches. A future CL will handle domains and entities.
Support is also added for filtering results by registrar.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170734664
Specifically, this prevents suspended registrars from creating domains or applications. Pending registrars already can't perform these actions because they get an error message when attempting to log in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170481338
This CL adds the functionality for nameserver searches. Future CLs will handle domains and entities.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170106014
There was a TODO to consider making RDAP queries look up domains by foreign key rather than directly to the domain itself. But since the TLD is a possible search filter, it makes more sense to stick with direct lookup, so we can take advantage of the index on TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169240318
When searching for domains by nameserver name or IP address, we fetch the matching nameserver keys, then search for domains by those keys. We limit the number of nameserver keys returned, to avoid arbitrarily large domain queries. This CL adds a warning to the RDAP response if we retrieved the maximum number of nameservers. This may indicate that we have not found all the domains.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168885124
We had been allowing lookups by nameserver name using a wildcard and suffix if the suffix was a domain name. That's ok if the domain name is one we manage, but doesn't work efficiently otherwise. A lookup of ns*.zombo.com would require us to search for all nameservers beginning with ns (which could be almost all of them), then loop through until we found those ending with .zombo.com. So we are going to prohibit suffixes after the TLD unless the suffix is an in-bailiwick domain.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168835732
RDAP filters some contact data when the request is not authenticated as coming from a user associated with the registrar owning the contact. This CL adds an exception for admin users, which for the App Engine Users API are defined as App Engine project viewers. This means that the registry team will always get all information when logged in. This will also be useful when building tools that use RDAP output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168293820
This CL changes the RDAP responses. When the requester asks for information about a domain, and is not logged in as the owning registrar, no contact information is shown. When the requester asks for information about a contact, and is not logged in as the owner registrar, the existence of the contact is shown, but not any personal data (the existence is shown to make things easier to test).
The login uses the same functionality as the registrar console.
For the most part, this CL does not include the necessary tests to make sure that data is not returned when not logged in. The CL is so large that I didn't want to burden it further. Those tests will be added in a follow-on CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168022034
