*** Reason for rollback ***
Going with a safer approach to using fresh poll message IDs that doesn't mutate domains themselves.
*** Original change description ***
Use PollMessage IDs that are globally unique across all time
The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.
***
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177081870
There is a big mix of different "load Resources" from different libraries
depending on where you were and what type of resource you want. Now there is a
clear hirarchy:
ResourceUtils:
for use in actual (non-test) code, reads a file from a context directory
TestDataHelper (uses ResourceUtils):
for use in tests, reads a file from a context directory + "/testdata". Also
caches the resource so calling it multiple times with the same file will
not read the file multiple times.
Library specific helpers (e.g. ToolsTestData) (uses TestDataHelper):
for use in that library's tests only, reads from a specific testdata directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177027533
Also, a couple tests are reworked to make their output more understandable in
case of failure.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176670087
This command is used by registry operators to apply registry locks to
domain names.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176577240
Dagger updated to 2.13, along with all its dependencies.
Also allows us to have multiple config files for different environment (prod, sandbox, alpha, local, etc) and specify which one to use on the command line with a --env flag. Therefore the same binary can be used in all environments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176551289
This command is used by registry operators to apply registry locks to
domain names.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176549874
The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176149220
Also changed truth.SubjectFactory to truth.Subject.Factory (plain renaming) and use method reference instead of anonymous class to create the factory when applicable.
FailureMetadata, an opaque object to its users, is introduced to replace FailureStrategy in in custom Subject in order to resolve some existing flaws of FailureStrategy as well as enable new features to be added to Truth.
New API is available in Truth-0.36, if there is a build/pom.xml, it's also updated to use this version.
More information: See []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176110788
Also, login logic pulled out to helper methods in the test class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
This serves as proof-of-concept to verify we can use Beam for our invoice generation use case. Namely, it checks that we can:
- Deploy a Beam template to GCS
- Read from Bigquery within the template
- Run the template from App Engine
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175755390
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.
The default wildcard type is now INVALID rather than NO_WILDCARD.
A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
The code to use this class will come in future CLs, to avoid a huge CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
This converts the upload task from a cron job to a task chained after staging.
This ensures the upload job only occurs when its dependencies are met, and
provides a faster turnaround time to verify both the staging and upload jobs
are complete.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175045489
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
This is the initial commit of the new billing system, rewritten as an Apache
Beam pipeline. This contains a basic end-to-end pipeline as proof of concept,
and boilerplate for GenerateInvoicesAction, which will eventually be our
automated invoice generation endpoint.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174184171
Also making these methods public so that other test methods can use them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174074038
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
I am not happy that another index is required, but the Pantheon console shows that domain indexes are much smaller than the other indexes (because there are fewer domains), so it's not adding an appreciable amount of storage space.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173561771
Currently to assert that a given Metric<Distribution> as a certain distribution for some labels, the caller needs to manually create an ImmutableDistribution and pass it to #hasValueForLabels method. With this change, an ImmutableSet of data points can be passed to #hasDataSetForLabels method.
Also switched to use expectThrow backport from JUnit 4.13.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173544521
This is a final refactor to address Nick's comments in [] where YearMonth really should be injected as a Joda type instead of a raw string, and the HTTP parameters should be separate from the default-provided dependencies.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173539965
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
JUnit 4.13 isn't released yet, but these functions are essential to being
able to write good test assertions about thrown exceptions. Rather than
not using them until JUnit 4.13 comes out (which might be awhile, as JUnit
4.12 came out almost three years ago), we're making the same decision that
Google made internally, which is to backport them. Indeed, the only reason
this commit is necessary is to fix breakage in the Nomulus build, as the
existing code worked fine internally where the backports are already in
place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173435579
I could've sworn we were already doing this, but apparently not? Anyway,
ROID suffixes have a number of requirements on them that weren't being
enforced, so this enforces them. All existing production data is compliant
with these requirements; the only existing bad data we have is in alpha and
sandbox.
ROID suffixes are now required to match the regex ^[A-Z0-9_]{1,8}$
See also https://tools.ietf.org/html/rfc5730
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173400001
This change:
- Adds retries to the staging action
- Emails domain-registry-eng@ upon completion of either action
- Simplifies logging to be more useful
TODO: fix up Module @Inject naming conventions and yearMonth injection
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173294822
We were relying on Dagger to validate the IP address, but that resulted in 500 errors when the IP address was not valid, which is undesirable. Instead, accept the parameters as strings, then convert them to IP addresses and throw a proper error when conversion fails.
Also fixes an improperly specified test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173172516
Usually, the correct order happens automatically, because we are searching on either the key or a specific field like fullyQualifiedDomainName, and the results come back in that order. But when searching by nameserver, where we split the query into multiple "IN" chunks, we have to assemble the result set and order after the fact.
The tests didn't pick up the problem, because the domains and hosts were created in alphabetical order, so it happened to work anyway. The tests have now been changed to create things in reverse order, to test the reordering. Also, the previous arbitrary limit of 1000 nameservers in the intermediate query has been reduced to 300, because we now loop through all nameservers no matter what, rather than stopping when we collect enough domains, so there's more of a penalty for having way too many nameservers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173163121
Adding the following metrics:
- how long does an update take, per TLD
- number of domains published, per TLD
- number of hosts published, per TLD
All are distributions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172933834
We no longer find contacts by name if the request is not authorized to see the name.
Several changes cascade from this. Previously, the code assumed that deleted contacts might still have full names, and therefore be searchable. This is not possible in all cases, because Datastore doesn't have the right index to find deleted contacts by name with a matching registrar. However, luckily, this situation can never occur, because contacts always have their name fields nulled out when they are deleted.
So instead, we simply ignore deleted records when searching by name, knowing that none can ever match.
The tests were then changed so that deleted records look the way the really will, meaning devoid of personal information.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172776926
This originally started as a small change, but quickly grew into a major refactor as I realized the original parameter structure wasn't conducive to a cron task and manual re-runs.
The changes are as follows:
1. Adds DNS metrics to activity reports, thanks to Nick's work with the Zoneman Dremel -> #plx workflow.
2. Surrounds registrar names in transactions reports with quotes, to escape possible commas.
3. Factors out the report generation logic into IcannReportingStager.
4. Assigns default values to the three main parameters
- yearMonth defaults to the previous month
- subdir defaults to "icann/monthly/yearMonth", i.e. "gs://domain-registry-reporting/icann/monthly/yyyy-MM"
- reportType defaults to both reports
5. Adds "Total" row generation logic to transactions reports
- This was a previously overlooked requirement.
6. Adds "MANIFEST.txt" generation and upload logic.
- The MANIFEST lists out which files need to be uploaded in the subdirectory.
7. Increases urlfetch timeout from 5s to 10s in backend tasks.
- Backend tasks should be more latency tolerant anyway, and this reduces the number of incorrect timeouts we see for services like Bigquery which might take some time to respond.
TESTED=Extensive testing in alpha, and ran FOSS test.
TODO: send out an e-mail for report generation and upload, and add reporting to cron.xml
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172738344
The TLD is added separately to the lock (the lock is unique per
TLD-ResourceName pair), so there's no need to add it to the resourceName.
The current status is that the TLD was included twice in the lockId. After the fix - it'll only be included once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172659986
This way it is consistent with the rest of our domain-related flows, which
consistently use the Domain* prefix. Note that claims checks are just a
special case of domain checks anyway, which run under DomainCheckFlow. This
will make dashboards looking at domain commands "just work" with a regexp of
Domain.*, without having to special-case in ClaimsCheck.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172608964
Also - remove logging from TransactNew, to prevent double logging on transient
failures (TransactNew retries on failure)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172500772
