The migration plan is as follows:
1. This CL, which adds the new "pubapi" service that serves the check API, WHOIS, and RDAP.
2a. Update our public facing sites to switch over to use the new service.
2b. (either order) Rewrite the check API to remove dependencies on flows.
3. ... eventually, once the frontend service is no longer being hit by this traffic, remove its handling of these public endpoints.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197716346
They were failing because the maximum App Engine task batch size is 1,000, and
we currently have more than 4,000 tasks in the pull queue. We keep re-uploading
those to NORDN because we're unable to delete the tasks after successful upload,
so the leases expire and they get processed again.
Also renames TaskEnqueuer to TaskQueueUtils to reflect its newly expanded role.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197060903
The issue is that the premium list cache is configured to persist for 60
minutes. So after updating the list, checks/creates for up to the next 60
minutes could still be referring to the old list. That's fine and dandy, unless
you delete the old premium list immediately (*bad*), which makes all domains
appear to now be non-premium for as long as the cache lasts. The reason deleting
the premium list entries makes names appear as non-premium is that a load-by-key
existence check with the domain label itself is used to determine if a name is
premium.
I also removed a misleading cache update statement, which doesn't do what it
appears to be doing (it appears to fix this issue) because cache is
instance-level, and so even if the premium list were updated from the frontend
instance only one of 100 instances would have its cache updated. But it's
updated from the tools service anyway, so it's guaranteed to not be a shared
cache with any instance serving EPP traffic.
On a sidenote, I introduced this bug on 2014-10-27 in [] The domain
label list refactor was my Noogler project.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197033604
Both WhoisAction and WhoisHttpAction set the HTTP response content type to "text/plain". There is no need to defensively escape the content. In fact, by escaping the content, it creates more problems down the line.
When used in a website, the response should be written into a DOM node by setting the textContent of the node, which automatically escapes the content.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196743398
This allows list_domains to continue working for large TLDs.
TESTED=Deploys to alpha and it works to list the most recently created domains even
on a TLD with a huge number of domains on it (much more than .app has currently).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196717389
Since the ConsoleReader now controls the display of the prompt, there is no
longer any way for an external program interacting with the nomulus shell to
recognize when the last command issued has been completed.
To remedy this, we introduce an "--encapsulate_output" flag, which causes
standard output and standard error to be wrapped in a class that precedes
all normal lines of output with a prefix ("out: " or "err: ", accordingly)
and allows the command processor to insert a "SUCCESS" or "FAILURE"
line (with no special prefix) after completing the processing of a
command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196702338
This claims data can exceed the maximum size of a task, causing the NORDN upload
to error out. It also wasn't even being used anyway. This data is already
logged during the upload and there's no reason to log it as well during the
verify, because there is already a unique actionLogId that can be used to tie
the verify task back to the upload task.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196521160
When we set the payload, it is converted to bytes using the response's
character set. Changing the MIME type later has no effect on the conversion
to bytes, even though it does change the returned MIME type. This results in
bytes that were encoded using one character set while the response reports a
different character set.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196258752
Jetty conflicts with appengine-integration-testing. Replaced one with the other.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196126483
Also added a couple of more tests to make sure that we cover all edge cases.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=195872013
The export happens in a reducer, whois instance fields all need to be serializable. The DriveConnection is therefore installed as a class variable that can be replaced with a mock during test. Class variables are not serialized. Only fields related to a particular instance are.
Note that DriveConnection is a misnomer. It is not a connection at all. It is just a thin wrapper class around the Drive service class, which provides convenient methods to write into Drive. Regardless, it cannot be serialized.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194596695
This efficiently counts domains without having to load them (as opposed to the
existing list_domains command which does load then).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194559095
The 'referralUrl' Datastore field is filled with mostly junk data, whereas
'url' contains real registrar web addresses. This makes the long needed fix to
display the proper url in WHOIS.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194398003
Currently we determine fee type from the fee extension description by checking if the format string of the FeeType contains the description we received. The formatting string for EAP is "Early Access Period, fee expires: %s", so the fee description generated by a domain check command, like "Early Access Period, fee expires: 2022-03-01T00:00:00.000Z", is not recognized as EAP.
This CL adds the ability to add arbitrary extra description strings to the FeeType for a description to match against. It also changes the match to "the given description contains any of the strings from the list of format string plus extra description strings".
For EAP, we added an extra description string "Early Access Period", so any fee description that contains "Early Access Period" will be matched to EAP FeeType, including the specific description (that contains the expiry time) that we send in a domain check.
Also improved error message on multiple fee type matching.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194149162
Per EPP RFC 5730, the <clTRID> element is optional. However, we weren't handling
it not being specified in asynchronous contact/host deletions because we were
adding it directly as a parameter value on a task, which does not allow null and
thus threw a NullPointerException.
This fixes handling for nulls (the parameter isn't set at all) and adds a test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194123259
This verifies anchor tenant creates by registrars are for exactly 2 years, to simplify billing down the line (anchor tenants get 2 years of free domain creates).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193931858
This adds a parameter to control invoice reporting, which defaults to false for
now (since we plan on manually adjusting the invoice next month).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193708323
(Usually the latter is because an IDE chose the repackaged name without the developer noticing.)
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193694213
This allows a registrar to check domain availability even if the fee extension is not set. To obtain the premium price or create the domain, the fee extension is still needed.
PS: I believe the previous tests are erroneous anyway. It is the presence of fee extensions in the check request that we were looking for. Checking if the extension is declared during login has no bearing on the results.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192778137
This hard-deletes all contacts and hosts owned by a specific set of registrar
client IDs, currently just "proxy".
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192325211
This makes them consistent with every other command, which uses the format
verb_noun.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192301468
This is in preparation for merging and then removing
RegistrarPremiumPriceAckAction.
This includes:
test that the data the UI sent isn't stale
---------------------------------------------
Our system is "read, modify, write". However, if between the "read" and the "write" someone else changed the registry, my write will undo their change even if I didn't touch any of their fields.
To solve that - we use the "lastUpdateTime" timestamp of the registrar. the UI reads it with the rest of the data, and sends it back on "write". We will now make sure the registrar currently in datastore has the same timestamp.
support premium-price-ack flag
---------------------------------
Add support for reading and writing this flag. We still won't be using it - that's in a followup CL, but we support it.
support changing the URL
------------------------
Add changing the URL in the UI, under the "whois" section
Will replace the Ack endpoint with this (and remove that endpoint) in a followup CL
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192154078
This is accomplished by making all non-mutating commands function with dry run set
to true, which also has the pleasurable side effect of not prompting for dry-run
mutating commands either, which also do nothing different/special on the second
run.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192149150
We don't want people to accidentally run commands on prod thinking they were on
Alpha / Sandbox.
To do that - we add 2 safeguards:
1) when on prod, the shell has a strong RED "PRODUCTION" in the commandline, while on alpha/sandbox it's green.
2) if a prod shell is idle for > 1h, it exits. So don't accidentally use a prod shell from a long time ago.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191931731
After writing a flag on the shell, pressing "tab" will print out the
documentation for that flag.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191899137
TestDataHelper is build exactly to prevent direct reads of resources. It caches
the resources and makes sure they are in the correct directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191785004
This allows us to not obtain a certificate and encrypt it with KMS when running the proxy locally during development.
Also updated FOSS build dagger version.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191746309
The registrar security console failed because it assumed the email is a
required field for the registrar, but it isn't (at least - create_registrar
doesn't require an email, and update_registrar lets you remove the email).
Fixed by allowing it to *remain* unset if it was unset originally, but if it was set - it's required.
There are more fixes needed, but they aren't related to the email, so they will wait for the next CL
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191623034
We no longer want to check end-date sunrise for OTE verification, so we should
remove any reference to allocation (and applications, which was removed in a
previous CL)
However, this last allocate was also the command that tested secDNS. So we need
to replace it with a regular create with secDNS.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191604222
This should prevent having issues with hot key paths on entities that
experience a heavy WHOIS volume (e.g. contacts that registrars reuse on
many domains).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191506124
Using the jline open-source library.
We save the history between invocations to ~/.nomulus_history
We add some simple completions:
- first argument completes to command name
- all other arguments complete to the command parameters, or filename
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191501023
Also increases the number of commit log buckets on alpha to 397 and correspondingly
reduces the frequency of commit log diff exporting to once every 3 minutes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191440586
JCommander doesn't seem to reset objects when it populates them with data from
an argument list during command processing, so recreate the command objects
every time we do a run().
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191332392
These servlets are converted to actions during daggerization. Calling them servers are misleading.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190942237
