Last commit did not pick up all the changes because MOE incorrectly attributed some changes to the wrong commit. This commit should reconcile these. Also picked up some changes to how hamcrest library is depended upon in BUILD file, which should have been included in previous commits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177637931
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
Also, a couple tests are reworked to make their output more understandable in
case of failure.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
Also, login logic pulled out to helper methods in the test class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.
The default wildcard type is now INVALID rather than NO_WILDCARD.
A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
The code to use this class will come in future CLs, to avoid a huge CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
I am not happy that another index is required, but the Pantheon console shows that domain indexes are much smaller than the other indexes (because there are fewer domains), so it's not adding an appreciable amount of storage space.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173561771
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
We were relying on Dagger to validate the IP address, but that resulted in 500 errors when the IP address was not valid, which is undesirable. Instead, accept the parameters as strings, then convert them to IP addresses and throw a proper error when conversion fails.
Also fixes an improperly specified test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173172516
Usually, the correct order happens automatically, because we are searching on either the key or a specific field like fullyQualifiedDomainName, and the results come back in that order. But when searching by nameserver, where we split the query into multiple "IN" chunks, we have to assemble the result set and order after the fact.
The tests didn't pick up the problem, because the domains and hosts were created in alphabetical order, so it happened to work anyway. The tests have now been changed to create things in reverse order, to test the reordering. Also, the previous arbitrary limit of 1000 nameservers in the intermediate query has been reduced to 300, because we now loop through all nameservers no matter what, rather than stopping when we collect enough domains, so there's more of a penalty for having way too many nameservers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173163121
We no longer find contacts by name if the request is not authorized to see the name.
Several changes cascade from this. Previously, the code assumed that deleted contacts might still have full names, and therefore be searchable. This is not possible in all cases, because Datastore doesn't have the right index to find deleted contacts by name with a matching registrar. However, luckily, this situation can never occur, because contacts always have their name fields nulled out when they are deleted.
So instead, we simply ignore deleted records when searching by name, knowing that none can ever match.
The tests were then changed so that deleted records look the way the really will, meaning devoid of personal information.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172776926
This CL adds the functionality for domain searches. Entities and nameservers have already been handled by previous CLs.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172097922
The nameserver may be external, in which case its TLD will not appear in our
list of valid TLDs, and the search will be rejected erroneously.
Tests for letter case canonicalizations also added at reviewer's suggestion.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171985702
Also sets the charset to UTF8 explicitly, to avoid confusion, and removes apparently unneeded Truth8 include.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171963981
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
Also fixed a bug discovered as a result of the more complete testing!
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171156304
This CL adds the functionality for contact and registrar searches. A future CL will handle domains and entities.
Support is also added for filtering results by registrar.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170734664
Specifically, this prevents suspended registrars from creating domains or applications. Pending registrars already can't perform these actions because they get an error message when attempting to log in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170481338
This CL adds the functionality for nameserver searches. Future CLs will handle domains and entities.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170106014
There was a TODO to consider making RDAP queries look up domains by foreign key rather than directly to the domain itself. But since the TLD is a possible search filter, it makes more sense to stick with direct lookup, so we can take advantage of the index on TLD.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169240318
When searching for domains by nameserver name or IP address, we fetch the matching nameserver keys, then search for domains by those keys. We limit the number of nameserver keys returned, to avoid arbitrarily large domain queries. This CL adds a warning to the RDAP response if we retrieved the maximum number of nameservers. This may indicate that we have not found all the domains.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168885124
We had been allowing lookups by nameserver name using a wildcard and suffix if the suffix was a domain name. That's ok if the domain name is one we manage, but doesn't work efficiently otherwise. A lookup of ns*.zombo.com would require us to search for all nameservers beginning with ns (which could be almost all of them), then loop through until we found those ending with .zombo.com. So we are going to prohibit suffixes after the TLD unless the suffix is an in-bailiwick domain.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168835732
RDAP filters some contact data when the request is not authenticated as coming from a user associated with the registrar owning the contact. This CL adds an exception for admin users, which for the App Engine Users API are defined as App Engine project viewers. This means that the registry team will always get all information when logged in. This will also be useful when building tools that use RDAP output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168293820
This CL changes the RDAP responses. When the requester asks for information about a domain, and is not logged in as the owning registrar, no contact information is shown. When the requester asks for information about a contact, and is not logged in as the owner registrar, the existence of the contact is shown, but not any personal data (the existence is shown to make things easier to test).
The login uses the same functionality as the registrar console.
For the most part, this CL does not include the necessary tests to make sure that data is not returned when not logged in. The CL is so large that I didn't want to burden it further. Those tests will be added in a follow-on CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168022034
To log autorenews, we currently run a mapreduce daily that creates synthetic
billing events for each recurring event past its due time. These are all
parented under the original recurring event, which allows these synthetic events to incorrectly stack on the original mutating entry.
We now explicitly create a new HistoryEntry of type DOMAIN_AUTORENEW to log
autorenews alongside other mutating EPP flows. These also parent DomainTransactionRecords for the NET_RENEWS_1_YEAR field, with the reporting time equal to the billing time (which accounts for the autorenew grace period).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166379700
We want to be safer and more explicit about the authentication needed by the many actions that exist.
As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
Up to now, our search wildcard rules have been that there must be an initial string of at least two characters. If a wildcard is present after that, it can optionally be followed by a suffix specifying the TLD (for domains) or domain (for nameservers). So domain queries can look like:
example.tld
ex*
ex*.tld
and nameserver queries can look like:
ns1.example.tld
ns*.example.tld
ns*
But you can't do a domain query for *.tld, nor a nameserver query for *.example.tld. It would be nice to support such queries, and the presence of a valid TLD or domain makes them relatively efficient. This CL relaxes the restrictions to allow wildcards with no initial string if the suffix is present. For nameservers, the suffix must be a valid domain in the system, to avoid having to loop through all nameservers.
A side effect of the changes is to fix a shortcoming in the logic which caused wildcard nameserver searches to fail if the specified domain suffix referred to an external domain.
Entity searches are not affected, since they do not support suffixes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159856563
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.
While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
In fact, completely eviscerate cloneProjectedAtTime (to be removed in
a followup CL) in favor of doing the projection of transfers and the
loading of values from the superordinate domain at call sites. This
is one of the issues that blocked the memcache audit work, since the
load inside of cloneProjectedAtTime could not be controlled by the
caller.
Note: fixed a minor bug where a subordinate host created after its superordinate domain was last transferred should have lastTransferTime==null but was previously reporting the domain's lastTransferTime.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149769125
A test has been added to RequestHandlerTest, making sure that, while we merely log errors for the time being, the correct dummy AuthResult is being created.
Most actions use the default settings, which have been changed to INTERNAL / APP / IGNORED. Actions with non-default settings are:
INTERNAL/NONE/PUBLIC (non-auth public endpoints)
CheckApiAction
WhoisHttpServer
Rdap*Action
INTERNAL,API/APP/ADMIN (things currently protected by web.xml)
EppTlsAction
EppToolAction
CreateGroupsAction
CreatePremiumListAction
DeleteEntityAction
List*sAction
UpdatePremiumListAction
VerifyOteAction
WhoisServer
INTERNAL,API,LEGACY/USER/PUBLIC (registrar console)
RegistrarPaymentAction
RegistrarPaymentSetupAction
RegistrarSettingsAction
EppConsoleAction
INTERNAL,API,LEGACY/NONE/PUBLIC (registrar console main page)
ConsoleUiAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149761652
LINKED is a virtual status that needs to be computed on the fly
when creating an RDAP response.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145583415
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072
We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018
A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55
The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.
Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:
- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
For some reason, although a constant was defined for the RDAP response content type, it was being used in one place but not another.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143185157
This is the second phase of a three phase migration to remove
ReferenceUnions. As of the end of this phase, ReferenceUnions are no longer read
from in any active code paths, but are still written to in case a rollback to
the previous version is necessary. The third and final phase will remove the
ReferenceUnions entirely.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137951076
