This is safer and addresses a common source of confusion in the codebase because it's always explicit that the resource returned may not be present, whether because it's soft-deleted when projected to the given time or because it never existed in the first place.
In production code, the presence of the returned value is always checked. In test code, its presence is assumed using .get() where that is expected and convenient, as it not being present will throw an NPE that will cause the test to fail anyway.
Note that the roughly equivalent reloadResourceByForeignKey(), which is widely used in test code, is not having this same treatment applied to it. That is out of the scope of this CL, and has much smaller returns anyway because it's only used in tests (where the unexpected absence of a given resource would just cause the test to fail).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225424002
Note that it's possible to set a config option to disable this functionality
on a per-environment basis (we're disabling it for sandbox), but in general
SSL certificate hashes should be required for increased security.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225053496
This will allow us to perform the OT&E history verification
in the model/ package as well so that it can be used both
by both the UI and the command line tool.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225007167
This is used to reduce the expiration time of domain(s) by some number of years
(if enough length remains in the registration term to do so). This does not back
out the previously saved BillingEvent entities as they may have already been
sent out and invoiced, so any related refunds must be handled out of band.
In addition to reducing the registration expiration time on the domain itself,
this command writes out a new history entry, one-time poll message informing the
registrar of this change, auto-renew billing event and poll message, and
updates/ends the old auto-renew billing event and poll message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224999285
EppResource's lastEppUpdateTime and lastEppUpdateClientId need
to be updated whenever the domain is updated, renewed, deleted or
transfered.
This commit applied the change to the following domain EPP commands:
- Update (already implemented)
- Renew
- Delete
- Restore
- Transfer request
- Transfer approve
- Transfer reject
- Transfer cancel
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223344758
Registrars that are PENDING or SUSPENDED should not have access to
EPP commands that cost money, as in either case it's not likely we'd
actually be able to get payment from said registrar. For this reason
we already prevented access to the domain create flow for non-active
registrars. This commit extends that to other commands that cost
money, including renewals, restores, and transfer requests.
Note that implicit autorenews will still occur for suspended
registrars, as in our point-in-time data model there's no good way
to prevent them. So when a registrar is suspended for non-payment,
the game plan is to get all of their domains transferred out to a
registrar that will pay as soon as possible.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223173677
EppConsoleAction still "manually" checks access by going over the
RegistrarContacts. We need it to use AuthenticatedRegistrarAccessor just like
every other part of the registrar console.
We still need to remove the (now unneeded) login EPP sent by the console, but that's left for a followup CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222404208
Instead of verifying interactions on the mocks, we instead assert on the real test subject directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218209556
Made it clear that it is a util class and moved some of the functions only used in NordnUploadAction (to NordnUploadAction). Also used Retrier to handle retries when leasing tasks.
These changes allow us to no longer use InjectRule in related unit tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217761117
This is an additional signal we can use for the billing/invoicing pipeline that
will be helfpul in targeting invoice rows that may need adjustment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217524250
The following issues are addressed:
- XML sanitizer should preserve input encoding. Gradle loses any that is not UTF-8. Bazel loses any that is not ASCII.
- Verify that XML sanitizer works with non-UTF8 encoding
- GpgSystemCommandRule breaks when $TMPDIR env variable is not set
- TestDataHelper throws exception when loading resources if resources are plain files on default file system as opposed to being in a jar file.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=216537258
These are simply too costly in their current form now that we are handling double-digit QPS, so at a minimum we'd want to refactor these for batched exports using a background thread (like how Stackdriver metrics work). However, upon further review, that work isn't worth doing if this BigQuery table isn't actually being used for anything, and it seems that we aren't using it anymore given that ICANN transaction reporting no longer requires it.
So the simplest thing to do is simply to get rid of this entirely, and just use a combination of Stackdriver metrics and App Engine logs. The eppMetrics BigQuery table is ~1.2 billion rows and takes up 223 GB, so that's not an insignificant GCP billings saving if we can delete it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=215905466
Note that the check flow does not yet handle any kind of allocation token
handling at all. Step 2 will be to add allocation token handling there, so a
RESERVED_FOR_ANCHOR_TENANT or RESERVED_FOR_SPECIFIC_USE domain will show as
available instead of reserved if the right token is specified using the
extension. Then once that's done, we can use that information to adjust the
price accordingly as well.
Right now the behavior with a domain check is that reserved domains always show
as reserved, even if they're anchor tenants.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=215599350
This is only useful when we used the [] proxy because the GFE requires SNI during handshake in order to request the client certificate. The GCP proxy does not need this (it always requests the client certificate). We do not need to check for its existence.
Also removed the checking of internal headers for ssl cert hash used only by the [] proxy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=213059027
It doesn't do anything that ALLOWED_IN_SUNRISE doesn't do, and there's no point
in having two separate types when we can simply keep track of the semantic
difference between the two by using different lists (as we have for .soy).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212997752
Unlike anchor tenants, these domains can be registered for any number of years,
but only during GA, as third parties cannot register domains pre-GA except
through the anchor tenant program.
Since this is new functionality, unlike creation of anchor tenants, there is no
fallback provided to send codes through the domain authcode; they must be sent
using the allocation token extension.
And note that, like with anchor tenants, providing the domain-specific
allocation token overrides any other reserved types that might apply to that
domain.
No changes are necessary to the domain application create flow because of the
above restriction to GA.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212310701
We never used it and don't have any plans to use it going forward. All
conceivable parts of its functionality that we might use going forward have
already been subsumed into allocation tokens, which are a simpler way of
handling the same use case that are also standards-compliant.
Also gets rid of the hideous ANCHOR_ prefix on anchor tenant EPP authcodes
that was only ever necessary because of overloading the authcode for
anchor tenant creation. Going forward it'll be based on allocation tokens,
so there's no risk of conflicts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209418194
Note that this gets rid of anchor tenant codes in reserved lists (yay!), which
are no longer valid. They have to come from allocation tokens now.
This removes support for LRP from domain application create flow (that's fine,
we never used it and I'm going to delete all of LRP later). It also uses
allocation tokens from EPP authcodes as a fallback, for now, but that will be
removed later once we switch fully to the allocation token mechanism.
This doesn't yet allow registration of RESERVED_FOR_SPECIFIC_USE domains using
the allocation token extension; that will come in the next CL. Ditto for
showing these reserved domains as available on domain checks when the allocation
token is specified.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209019617
Masks user credentials (tags 'pw' and 'newPW') in EPP XML messages.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207953894
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).
This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.
This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
Domains that are reserved with type NAME_COLLISION can be registered defensively
during sunrise only, but DNS can never resolve for them. Correspondingly, we
need to apply the SERVER_HOLD status for such registrations. We also send the
registrar a poll message informing them of this act.
This brings us up to feature parity with end-date sunrise (implemented in
DomainAllocateFlow), which already has all of this handling.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205277728
Async tasks will now re-enqueue themselves after completion if there are
additional pending future actions. This allows all parts of domain delete flows
to be successfully re-saved as the parts happen, without going past the maximum
allowed 30 day task ETA limit. The first task runs at 30 days out when the
redemption grace period ends, and that task then enqueues another task to run 5
more days in the future, when the deletion is final and the pending delete
status gets removed.
No data migration plan is necessary because future resaves defaults to empty,
and indeed will always be empty on transfers. So previously enqueued tasks will
still be valid.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202949677
This affects JSR305, JSR330, and Guava annotations.
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
We'll continue to use injected clocks for the rest of our tests that use signed marks files, so that they don't all fail after the current validity period. The new test TmchTestDataExpirationTest will let us know when the files are expired, so we can update them.
All updated test data files come from https://newgtlds.icann.org/en/about/trademark-clearinghouse/registries-registrars
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202208196
In RFC 5730, clTrid is specified as optional. We ran into an error earlier this
year in which a registrar was not passing a client transaction id and we didn't
handle it correctly. So, this CL adds some tests of common EPP operations verify
that they work correctly when the clTrid is not specified.
This also slightly improves some flow logic to make it more obvious at first
glance that clTrid is indeed optional.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202000845
This is used in the domain transfer and delete flows, both of which are
asynchronous flows that have implicit default actions that will be taken at some
point in the future. This CL adds scheduled re-saves to take place soon after
those default actions would become effective, so that they can be re-saved
quickly if so.
Unfortunately the redemption grace period on our TLDs is 35 days, which exceeds
the 30 day maximum task ETA in App Engine, so these won't actually fire. That's
fine though; the deletion is actually effective as of 5 days, and this is just
removing the grace period.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201345274
Now that the large zone re-signing test is complete, we no longer need it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
This will allow us to check in actual SUNRISE billing policies per launch (15% discount), instead of relying on ad-hoc timestamps.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200077926
Currently the input XML to an EPP flow is logged twice, once
in FlowRunner and once in FlowReporter.
The log by FlowReporter was used by reporting but this is no
longer the case.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=200057153
Also prevents signed marks from being used in non-sunrise TldStates.
Currently, we send out a Lordn update only when there's a ClaimNotice, or if
we're in end-date sunrise.
But EPPs can contain a SignedMark instead of a ClaimsNotice for trademarked
domains - in which case we aren't sending out Lordn update. This also applies
to start-date sunrises.
We also change the SignedMark behavior for superusers. Currently, if a
mismatched signed mark is given as superuser, we accept it. That causes
problems when we want to send the Lordn update.
Instead - we no longer allow superusers to give a mismatched SignedMark (just
as we don't allow users to give a bad ClaimNotice). A super user can still
create a domain WITHOUT a signed mark - but if one is provided, it must match.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199783411
New metrics are necessary because the new API no longer wraps
an EPP flow, therefore does not get metrics for free.
Metrics include
- An EventMetric for processing time
- An IncrementableMetric for request count, with
availability (available/reserved/registered) and
pricing (standard/premium) fields
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199708592
This is consistent with how we treat RESTORE billing events as well- in
general, fees are considered to be amortized over the course of a year (by the
invoicing team).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199684843
Superuser can also execute contact info commands. AuthInfo is no longer checked in the input and always displayed in the output as the only ones who can get a response are the sponsoring registrar and super user.
Also corrected a Javadoc in which '@' should have been escaped (see https://reflectoring.io/howto-format-code-snippets-in-javadoc/)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199521153
Copied class and test from CheckApiAction. All unit tests passing.
Remaining work: add metrics
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198916177
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
We've moved completely to the JSON based reporting framework. The legacy logging statement is only for human consumption, therefore removing the comments. Also fixes a bug where the last argument is not used due to the formatter only expecting 7 arguments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198558998
When we set the payload, it is converted to bytes using the response's
character set. Changing the MIME type later has no effect on the conversion
to bytes, even though it does change the returned MIME type. This results in
bytes that were encoded using one character set while the response reports a
different character set.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196258752
Also added a couple of more tests to make sure that we cover all edge cases.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195872013
serves from https now
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195473725
Currently we determine fee type from the fee extension description by checking if the format string of the FeeType contains the description we received. The formatting string for EAP is "Early Access Period, fee expires: %s", so the fee description generated by a domain check command, like "Early Access Period, fee expires: 2022-03-01T00:00:00.000Z", is not recognized as EAP.
This CL adds the ability to add arbitrary extra description strings to the FeeType for a description to match against. It also changes the match to "the given description contains any of the strings from the list of format string plus extra description strings".
For EAP, we added an extra description string "Early Access Period", so any fee description that contains "Early Access Period" will be matched to EAP FeeType, including the specific description (that contains the expiry time) that we send in a domain check.
Also improved error message on multiple fee type matching.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194149162
Per EPP RFC 5730, the <clTRID> element is optional. However, we weren't handling
it not being specified in asynchronous contact/host deletions because we were
adding it directly as a parameter value on a task, which does not allow null and
thus threw a NullPointerException.
This fixes handling for nulls (the parameter isn't set at all) and adds a test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194123259
This verifies anchor tenant creates by registrars are for exactly 2 years, to simplify billing down the line (anchor tenants get 2 years of free domain creates).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193931858
This allows a registrar to check domain availability even if the fee extension is not set. To obtain the premium price or create the domain, the fee extension is still needed.
PS: I believe the previous tests are erroneous anyway. It is the presence of fee extensions in the check request that we were looking for. Checking if the extension is declared during login has no bearing on the results.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192778137
