This is a 'red' Flogger migration CL. Red CLs contain changes which are
likely not to work without manual intervention.
Note that it may not even be possible to directly migrate the logger
usage in this CL to the Flogger API and some additional refactoring may
be required. If this is the case, please note that it should be safe to
submit any outstanding 'green' and 'yellow' CLs prior to tackling this.
If you feel that your use case is not covered by the existing Flogger API
please raise a feature request at []and
revert this CL.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198463651
This is a 'yellow' Flogger migration CL. Yellow CLs should be mostly safe
but include changes that are notable for one reason or another. Manual
intervention may be required to address small issues.
The comments in this CL indicate cases where suggested code changes
should be double checked, or even modified. There may even be cases where
files outside this CL are affected by changes to things such as logger
visibility. However if a change does not have an associated comment then
it should be safe.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097990
I'm tired of running check_domain only to then realize I needed to run
check_domain_fee instead because the domain ended up being premium. We require
the use of the fee extension to register domains on all of our TLDs anyway so we
might as well always be sending the fee extension when checking domains
too. There's no additional harm in sending along the fee extension (it only
sends back additional information that is often useful), so just make that the
default and remove the extra unnecessary command.
Note that check_domain_claims can't be merged in too because it fundamentally
works differently. It doesn't query the availability of domain names for
registration, just whether they're trademarked.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198097326
We've already had the need to do this on a few occasions, so it's best to make
it easy rather than requiring hand-generated XML all the time.
Also normalizes the boolean --registrar_request parameter to not have arity=1.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198093829
You don't want to use the cache when loading them for the purposes of updating
them, but you definitely do still want to use the cache when checking the
price of individual domains.
In [] the cache clearing of premium lists on update was removed. This
is a good thing in aggregate because the cache is per-instance and thus
misleading, but it also caused us to not be able to update the same premium
list twice within an hour because the second update would hit a "PremiumList
was concurrently edited" exception, owing to first loading the stale version
from the cache for the purposes of updating it. Now we bypass the cache for
that purpose.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197768142
The migration plan is as follows:
1. This CL, which adds the new "pubapi" service that serves the check API, WHOIS, and RDAP.
2a. Update our public facing sites to switch over to use the new service.
2b. (either order) Rewrite the check API to remove dependencies on flows.
3. ... eventually, once the frontend service is no longer being hit by this traffic, remove its handling of these public endpoints.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197716346
This is a 'red' Flogger migration CL. Red CLs contain changes which are
likely not to work without manual intervention.
Note that it may not even be possible to directly migrate the logger
usage in this CL to the Flogger API and some additional refactoring may
be required. If this is the case, please note that it should be safe to
submit any outstanding 'green' and 'yellow' CLs prior to tackling this.
If you feel that your use case is not covered by the existing Flogger API
please raise a feature request at []and
revert this CL.
For more information, see []
Base CL: 197331037
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197503952
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197331037
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197466715
This adds a dummy flogger logging statement in the GCP proxy to ensure that it
works.
TESTED=Deployed to alpha and verified that flogger works. Also passed FOSS
tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196899036
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
This makes it easier to debug issues. There are no privacy concerns in sandbox.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197045576
The issue is that the premium list cache is configured to persist for 60
minutes. So after updating the list, checks/creates for up to the next 60
minutes could still be referring to the old list. That's fine and dandy, unless
you delete the old premium list immediately (*bad*), which makes all domains
appear to now be non-premium for as long as the cache lasts. The reason deleting
the premium list entries makes names appear as non-premium is that a load-by-key
existence check with the domain label itself is used to determine if a name is
premium.
I also removed a misleading cache update statement, which doesn't do what it
appears to be doing (it appears to fix this issue) because cache is
instance-level, and so even if the premium list were updated from the frontend
instance only one of 100 instances would have its cache updated. But it's
updated from the tools service anyway, so it's guaranteed to not be a shared
cache with any instance serving EPP traffic.
On a sidenote, I introduced this bug on 2014-10-27 in [] The domain
label list refactor was my Noogler project.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197033604
When EPP SSL handshake is unsuccessful, #channelInactive is called but there are no quotas to return, because quotas are only leased upon the first #channelRead. There is no need to log a warning and throw an exception in this case because the handshake exception would have been thrown already. Throwing a second exception just crowds the log.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197016756
This adds a dummy flogger logging statement in the GCP proxy to ensure that it
works.
TESTED=Deployed to alpha and verified that flogger works. Also passed FOSS
tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196899036
100 is way overkill with manual scaling. 30 is most likely still overkill too,
but we want to tune incrementally rather than all at once. Note that at 30
instances we're expecting around 3 QPS per instance, which is still an order
of magnitude less than each instance can actually handle.
This also fixes the instance type on sandbox to be the same as on prod.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196875876
Gustavo Lozano confirmed that street addresses should be formatted as a string
if there is only one line, or an nested array of strings if there is more than
one. Luckily we already do it this way, so we just need to update the comment.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196845222
Both WhoisAction and WhoisHttpAction set the HTTP response content type to "text/plain". There is no need to defensively escape the content. In fact, by escaping the content, it creates more problems down the line.
When used in a website, the response should be written into a DOM node by setting the textContent of the node, which automatically escapes the content.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196743398
It turns out that SourceCodeEscapers has still not been released to github.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196721579
This allows list_domains to continue working for large TLDs.
TESTED=Deploys to alpha and it works to list the most recently created domains even
on a TLD with a huge number of domains on it (much more than .app has currently).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196717389
Since the ConsoleReader now controls the display of the prompt, there is no
longer any way for an external program interacting with the nomulus shell to
recognize when the last command issued has been completed.
To remedy this, we introduce an "--encapsulate_output" flag, which causes
standard output and standard error to be wrapped in a class that precedes
all normal lines of output with a prefix ("out: " or "err: ", accordingly)
and allows the command processor to insert a "SUCCESS" or "FAILURE"
line (with no special prefix) after completing the processing of a
command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196702338
This claims data can exceed the maximum size of a task, causing the NORDN upload
to error out. It also wasn't even being used anyway. This data is already
logged during the upload and there's no reason to log it as well during the
verify, because there is already a unique actionLogId that can be used to tie
the verify task back to the upload task.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196521160
When we set the payload, it is converted to bytes using the response's
character set. Changing the MIME type later has no effect on the conversion
to bytes, even though it does change the returned MIME type. This results in
bytes that were encoded using one character set while the response reports a
different character set.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196258752
Also added a couple of more tests to make sure that we cover all edge cases.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195872013
serves from https now
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195473725
This should decrease the average wait time when running nomulus tool.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195465469
Increase the instances on alpha to achieve parity with sandbox.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194980588
The export happens in a reducer, whois instance fields all need to be serializable. The DriveConnection is therefore installed as a class variable that can be replaced with a mock during test. Class variables are not serialized. Only fields related to a particular instance are.
Note that DriveConnection is a misnomer. It is not a connection at all. It is just a thin wrapper class around the Drive service class, which provides convenient methods to write into Drive. Regardless, it cannot be serialized.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194596695
This efficiently counts domains without having to load them (as opposed to the
existing list_domains command which does load then).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194559095
It doesn't make sense to log all 10K+ lines of a premium list every time it's
updated, and indeed that seems to hurt performance, yet that's precisely what
we were doing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194449836
The 'referralUrl' Datastore field is filled with mostly junk data, whereas
'url' contains real registrar web addresses. This makes the long needed fix to
display the proper url in WHOIS.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194398003
ListObjectsCommand is a base class for a number of List commands that query the
tools server with the command input, and then show the user the server's reply.
For example, ListDomainsCommand queries ListDomainsAction with the input TLDs
from the user, and then prints the resulting domains for the user.
Currently, when the server query returns an error (because, e.g., the user of
ListDomainsCommand gave a non-existing TLD) - ListObjectsCommand just prints
the error message to stdout, and returns successfully.
That means that any automatic tool would think the command succeeded.
This CL changes the behavior of ListObjectsCommand to throw an exception with
the error message if the server returned an error. This results in the nomulus
command failing if the server encountered an error.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194393053
Currently we determine fee type from the fee extension description by checking if the format string of the FeeType contains the description we received. The formatting string for EAP is "Early Access Period, fee expires: %s", so the fee description generated by a domain check command, like "Early Access Period, fee expires: 2022-03-01T00:00:00.000Z", is not recognized as EAP.
This CL adds the ability to add arbitrary extra description strings to the FeeType for a description to match against. It also changes the match to "the given description contains any of the strings from the list of format string plus extra description strings".
For EAP, we added an extra description string "Early Access Period", so any fee description that contains "Early Access Period" will be matched to EAP FeeType, including the specific description (that contains the expiry time) that we send in a domain check.
Also improved error message on multiple fee type matching.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194149162
Per EPP RFC 5730, the <clTRID> element is optional. However, we weren't handling
it not being specified in asynchronous contact/host deletions because we were
adding it directly as a parameter value on a task, which does not allow null and
thus threw a NullPointerException.
This fixes handling for nulls (the parameter isn't set at all) and adds a test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194123259
This verifies anchor tenant creates by registrars are for exactly 2 years, to simplify billing down the line (anchor tenants get 2 years of free domain creates).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193931858
This adds a parameter to control invoice reporting, which defaults to false for
now (since we plan on manually adjusting the invoice next month).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193708323
This makes it more clear than an operation is actually pending, rather
than nothing changing after pressing "Y" to run the command.
The prompt now looks like when you press "Y" and "N", respectively:
Perform this command? (y/N): Running ...
Perform this command? (y/N): Command aborted.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193521084
Locales are weird. Even if all our character individually are just 0-9a-z_,
different locales might still convert them differently to upper/lower cases...
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193512312
