The previous functionality was reusing the same PollMessage ID for Autorenews
every year. This can potentially cause confusion at registrars if they were
expecting these to be globall unique across all time. So this change simply
changes the ID during autorenew.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176149220
Also changed truth.SubjectFactory to truth.Subject.Factory (plain renaming) and use method reference instead of anonymous class to create the factory when applicable.
FailureMetadata, an opaque object to its users, is introduced to replace FailureStrategy in in custom Subject in order to resolve some existing flaws of FailureStrategy as well as enable new features to be added to Truth.
New API is available in Truth-0.36, if there is a build/pom.xml, it's also updated to use this version.
More information: See []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176110788
Also, login logic pulled out to helper methods in the test class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175870131
This CL also includes a minor refactor of the query size limit calculation; it is computed and stored in a local variable, to be used in two places, rather than computing it separately in each place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175824713
This serves as proof-of-concept to verify we can use Beam for our invoice generation use case. Namely, it checks that we can:
- Deploy a Beam template to GCS
- Read from Bigquery within the template
- Run the template from App Engine
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175755390
This CL uses the previously-defined RDAP metrics class to record basic metrics
for all RDAP endpoints, and handles testing of non-search endpoints. Searches
are more complicated, and will be handled in future CLs.
The default wildcard type is now INVALID rather than NO_WILDCARD.
A change to getMatchingResources() (adding an additional parameter) is also included in this CL, as it was needed to set the incompleteness warning type correctly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175719265
The code to use this class will come in future CLs, to avoid a huge CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175162557
This converts the upload task from a cron job to a task chained after staging.
This ensures the upload job only occurs when its dependencies are met, and
provides a faster turnaround time to verify both the staging and upload jobs
are complete.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175045489
I split this out to avoid having a giant CL that changes everything. The actual
metrics will follow later.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174356874
This is the initial commit of the new billing system, rewritten as an Apache
Beam pipeline. This contains a basic end-to-end pipeline as proof of concept,
and boilerplate for GenerateInvoicesAction, which will eventually be our
automated invoice generation endpoint.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174184171
Also making these methods public so that other test methods can use them.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=174074038
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
I am not happy that another index is required, but the Pantheon console shows that domain indexes are much smaller than the other indexes (because there are fewer domains), so it's not adding an appreciable amount of storage space.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173561771
Currently to assert that a given Metric<Distribution> as a certain distribution for some labels, the caller needs to manually create an ImmutableDistribution and pass it to #hasValueForLabels method. With this change, an ImmutableSet of data points can be passed to #hasDataSetForLabels method.
Also switched to use expectThrow backport from JUnit 4.13.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173544521
This is a final refactor to address Nick's comments in [] where YearMonth really should be injected as a Joda type instead of a raw string, and the HTTP parameters should be separate from the default-provided dependencies.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173539965
They can be inferred correctly even in Java 7, and display as
compiler warnings in IntelliJ.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173451087
JUnit 4.13 isn't released yet, but these functions are essential to being
able to write good test assertions about thrown exceptions. Rather than
not using them until JUnit 4.13 comes out (which might be awhile, as JUnit
4.12 came out almost three years ago), we're making the same decision that
Google made internally, which is to backport them. Indeed, the only reason
this commit is necessary is to fix breakage in the Nomulus build, as the
existing code worked fine internally where the backports are already in
place.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173435579
I could've sworn we were already doing this, but apparently not? Anyway,
ROID suffixes have a number of requirements on them that weren't being
enforced, so this enforces them. All existing production data is compliant
with these requirements; the only existing bad data we have is in alpha and
sandbox.
ROID suffixes are now required to match the regex ^[A-Z0-9_]{1,8}$
See also https://tools.ietf.org/html/rfc5730
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173400001
This change:
- Adds retries to the staging action
- Emails domain-registry-eng@ upon completion of either action
- Simplifies logging to be more useful
TODO: fix up Module @Inject naming conventions and yearMonth injection
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173294822
We were relying on Dagger to validate the IP address, but that resulted in 500 errors when the IP address was not valid, which is undesirable. Instead, accept the parameters as strings, then convert them to IP addresses and throw a proper error when conversion fails.
Also fixes an improperly specified test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173172516
Usually, the correct order happens automatically, because we are searching on either the key or a specific field like fullyQualifiedDomainName, and the results come back in that order. But when searching by nameserver, where we split the query into multiple "IN" chunks, we have to assemble the result set and order after the fact.
The tests didn't pick up the problem, because the domains and hosts were created in alphabetical order, so it happened to work anyway. The tests have now been changed to create things in reverse order, to test the reordering. Also, the previous arbitrary limit of 1000 nameservers in the intermediate query has been reduced to 300, because we now loop through all nameservers no matter what, rather than stopping when we collect enough domains, so there's more of a penalty for having way too many nameservers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173163121
Adding the following metrics:
- how long does an update take, per TLD
- number of domains published, per TLD
- number of hosts published, per TLD
All are distributions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172933834
We no longer find contacts by name if the request is not authorized to see the name.
Several changes cascade from this. Previously, the code assumed that deleted contacts might still have full names, and therefore be searchable. This is not possible in all cases, because Datastore doesn't have the right index to find deleted contacts by name with a matching registrar. However, luckily, this situation can never occur, because contacts always have their name fields nulled out when they are deleted.
So instead, we simply ignore deleted records when searching by name, knowing that none can ever match.
The tests were then changed so that deleted records look the way the really will, meaning devoid of personal information.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172776926
This originally started as a small change, but quickly grew into a major refactor as I realized the original parameter structure wasn't conducive to a cron task and manual re-runs.
The changes are as follows:
1. Adds DNS metrics to activity reports, thanks to Nick's work with the Zoneman Dremel -> #plx workflow.
2. Surrounds registrar names in transactions reports with quotes, to escape possible commas.
3. Factors out the report generation logic into IcannReportingStager.
4. Assigns default values to the three main parameters
- yearMonth defaults to the previous month
- subdir defaults to "icann/monthly/yearMonth", i.e. "gs://domain-registry-reporting/icann/monthly/yyyy-MM"
- reportType defaults to both reports
5. Adds "Total" row generation logic to transactions reports
- This was a previously overlooked requirement.
6. Adds "MANIFEST.txt" generation and upload logic.
- The MANIFEST lists out which files need to be uploaded in the subdirectory.
7. Increases urlfetch timeout from 5s to 10s in backend tasks.
- Backend tasks should be more latency tolerant anyway, and this reduces the number of incorrect timeouts we see for services like Bigquery which might take some time to respond.
TESTED=Extensive testing in alpha, and ran FOSS test.
TODO: send out an e-mail for report generation and upload, and add reporting to cron.xml
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172738344
The TLD is added separately to the lock (the lock is unique per
TLD-ResourceName pair), so there's no need to add it to the resourceName.
The current status is that the TLD was included twice in the lockId. After the fix - it'll only be included once.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172659986
This way it is consistent with the rest of our domain-related flows, which
consistently use the Domain* prefix. Note that claims checks are just a
special case of domain checks anyway, which run under DomainCheckFlow. This
will make dashboards looking at domain commands "just work" with a regexp of
Domain.*, without having to special-case in ClaimsCheck.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172608964
Also - remove logging from TransactNew, to prevent double logging on transient
failures (TransactNew retries on failure)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172500772
This makes the units explicit, which prevents confusion and bugs.
More information: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172455602
The code was not checking the value returned by getTrid() to make sure it was
not null.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172384499
This will result is a prompt like
Creating TLD with:
projectId= domain-registry-alpha
description= some description
dnsName= mytld.
name= mytld.
nameServerSet= cloud-dns-registry-test
Perform this command? (y/N):
before actually performing the command, and adds a --force flag to bypass the
prompt.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172369944
This CL adds the functionality for domain searches. Entities and nameservers have already been handled by previous CLs.
Deleted items can only be seen by admins, and by registrars viewing their own deleted items.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172097922
RDAP searches for contacts with a specific desired registrar need an additional
index term. The tests were not extensive enough to catch this particular case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172013843
Convert periods to hyphens in multi-part TLDs when using them as a zone name
(cloud-dns doesn't allow periods in zone names).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172007089
Unfortunately this tool isn't smart enough to deal with the assertThat
situation (which has two static imports of a function with the same name).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172000753
The nameserver may be external, in which case its TLD will not appear in our
list of valid TLDs, and the search will be rejected erroneously.
Tests for letter case canonicalizations also added at reviewer's suggestion.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171985702
Also sets the charset to UTF8 explicitly, to avoid confusion, and removes apparently unneeded Truth8 include.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171963981
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
This CL adds transferredRegistrationExpirationTime as a TransferData field
persisted to Datastore. It's only relevant for domains, and it represents the
registration expiration time resulting from the approval of the most recent
transfer request. For pending transfers, we assume the transfer will be
server-approved, and thus in DomainTransferRequestFlow we set this field to the
existing computed value serverApproveNewExpirationTime, which is what we use
for setting up the server-approve autorenew billing event and poll message.
In DomainTransferApproveFlow we overwrite this field with the freshly computed
newExpirationTime, whereas in DomainTransferCancel/RejectFlow (and in the
implicit cancel of DomainDeleteFlow during a pending transfer) we null it out.
There are two key benefits to having this field, which are described in more
detail in b/36405140.
1) b/25084229 - it allows storage of a frozen value to back the "exDate" field
of DomainTransferResponse, which we can use to fix various errors with how
exDate display currently works.
2) b/36354434 - it allows DomainResource.cloneProjectedAtTime() to just directly
set the registrationExpirationTime to this value, without computing it de
novo, which reduces duplicated logic and ensures that the new expiration time
matches the autorenew child objects.
This CL only starts writing the field on TransferData as persisted directly on
the DomainResource itself. We'll then want to backfill the field for at
least pending transfers, whether expired or not (so we can do (2) above), but
I think we might as well backfill it for all pending and approved transfers
so that we also fix (1) even for historical transfers. And then we can start
actually reading the field for both purposes. (Note that for (1), this will
only fix synchronous transfer responses served via DomainTransferQueryFlow,
not async transfer responses served via poll messages, since these have already
been persisted with a potentially bad exDate, but I don't think it's worth a
backfill for those).
One last naming note: I chose the verbose transferredRegistrationExpirationTime
rather than the extendedRegistrationExpirationTime of DomainTransferResponse
because (as is the case in autorenew grace, or for a superuser transfer) the
new registration time isn't necessarily extended at all; it may be the same as
the pre-transfer expiration time. Also, including "registration" helps clarify
w.r.t. pendingTransferExpirationTime which refers confusingly to the expiry of
the transfer itself, rather than the domain registration.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171858083
