Whenever a registrar is changed via the registrar console, we send out a
notification of that change.
Since we're going to allow Admins and soon Vendors to use the console in
addition to the registrars, it becomes important to know who actually performed
the changes if the registrars complain.
In addition, we will now send notifications for changes in Sandbox since we're
going to actually allow registrars to update sandbox data.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217539534
All credentials provided by these modules have been
replaced by those in the config/CredentialsModule,
with a new set of Qualifiers. With Dagger 2, a successful
build means that the removal is safe.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=215258792
RegistrarSettingsAction is a JSON in / JSON out endpoint, meaning the reply is consumed as JSON.
The current state is that if an error occurs, there are two possible replies:
- a JSON error reply is sent out, or
- a 402 HTML reply is sent out with the exception.getMessage()
The difference is only - do we actively catch the exception to translate it to JSON or not.
This fix catches ALL exceptions and translates them to JSON format. Note that there's no security change by giving the getMessage in the JSON reply since we were returning that anyway (in the HTML).
In addition - changed the "gaeUserId" to "user.getEmail" as the identifier, since it's clearer to the users who see that error - and I do want to transition to a more "email identifier" way of checking access (since that's what users put in the registrar contact info)
This too isn't leaking new information because
- the initial HTML page load already gives the user's email, and
- the logs already log the user's email for every request
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=215213807
Replaced the plethora of inter winding access functions and inputs in SessionUtils with just 2 functions, that both accept the same type for the user (AuthResult):
guessRegistrarForUser: given an AuthResult, finds a registrar that they have access to. If none is found - a ForbiddenException is thrown.
getRegistrarForUser[Cached]: (maybe should be called getRegistrarOnBehalfOfUser?) given an AuthResult and a clientId, loads and returns the registrar ONLY IF the user has access to it. Otherwise throws a ForbiddenException.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=214630657
Updated the registar contact group management, which is the only
use case for this credential.
Also updated GSuite domain delegated admin access config in admin
dashboard for both sandbox (used by alpha and sandbox) and prod.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212320157
ModulesService does not provide a great API. Specifically, it doesn't have a
way to get the hostname for a specific service; you have to get the hostname for
a specific version as well. This is very rarely what we want, as we publish new
versions every week and don't expect old ones to hang around for very long, so
a task should execute against whatever the live version is, not whatever the
current version was back when the task was enqueued (especially because that
version might be deleted by now).
This new and improved wrapper API removes the confusion and plays better with
dependency injection to boot. We can also fold in other methods having to do
with App Engine services, whereas ModulesService was quite limited in scope.
This also has the side effect of fixing ResaveEntityAction, which is
currently broken because the tasks it's enqueuing to execute up to 30 days in
the future have the version hard-coded into the hostname, and we typically
delete old versions sooner than that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206173763
This ensures that only one will run at a time, which should help fix the
clogged up mapreduces we've seen on sandbox.
In order to do this, the UnlockerOutput is introduced. This unlocks the
given Lock after all reducer shards have finished.
Also increases the lease duration of the DNS refresh action from 20 to
240 minutes. 20 minutes isn't long enough; when there's a lot of domains
and decent system load the mapreduce could take longer than that in the
ordinary case.
TESTED=Deployed to alpha and verified that more than one copy of the
mapreduce wouldn't run simultaneously, and also that the lock is
released when the mapreduce is finished.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205887554
This affects JSR305, JSR330, and Guava annotations.
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
It doesn't entirely make semantic sense, since the actual state of the
SystemClock isn't being preserved, but it makes injection into serializable
classes (e.g. mapreduces) much simpler, so it's worth doing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201755949
SystemClock isn't Serializable (for obvious reasons), whereas LockHandlerImpl is
used as a field on some Serializable mapreduce classes. So mark it transient and
then re-generate it on first use following de-serialization when it happens to
be null.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201707209
Now that the large zone re-signing test is complete, we no longer need it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
Rather than just logging a generic TimeoutException, this will say what action
timed out and how long it had been executing for.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201049025
Currently, we have two different ways to parse a "set" parameter:
key=value1&key=value2&key=value3...
and
keys=value1,value2,value3
This is error prone for several reasons:
- different parts of the code must be "synchronized" to use the same style (the
place that creates the request, and the place that parses the request)
- for the key=value1&key=value2, we often use the same key name for the single
value and the set value. This can result in subtle bugs where part of the
code will successfully read the key assuming there's only one key (and will
get the first key=value1, ignoring the rest)
Here we transition everything to the keys=value1,value2,value3 method. This one
was chosen because:
- it's shorter
- it's more intuitive for users
- the key name is plural, differentiating it from the singular key=value that
other requests might need
-----------------------------------
To make sure there are not "transition issues", we will continue to support
(with warnings) the key=value1&key=value2 parameter parsing until we're sure we
haven't forgotten to update any part of the code.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198810681
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
where the value will be immediately unboxed anyway.
The change removes small-but-pervasive inefficiencies from creating and
immediately discarding instances of the wrapped value, as well as removing
unnecessary syntax.
More information: []
Tested:
TAP --sample for global presubmit queue
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189704431
This moves the default yearMonth logic into a common ReportingModule, rather than the coarse-scoped BackendModule, which may not want the default parameter extraction logic, as well as moving the 'yearMonth' parameter constant to the common package it's used in. This also provides a basis for future consolidation of the ReportingEmailUtils and BillingEmailUtils classes, which have modest overlap.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=183130311
NotModifiedException was using HttpServletResponse.SC_NOT_FOUND instead of SC_NOT_MODIFIED (likely an autocomplete typo).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182976671
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
Most of the time, we don't expect incoming requests to have an authorization
header. So this statement gets printed a lot, and doesn't provide much useful
information. We already have a statement listing what type of
authentication/authorization is required by the endpoint, and other statements
indicating either that authorization was successful with a particular method or
was not successful at all.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175969652
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
We were relying on Dagger to validate the IP address, but that resulted in 500 errors when the IP address was not valid, which is undesirable. Instead, accept the parameters as strings, then convert them to IP addresses and throw a proper error when conversion fails.
Also fixes an improperly specified test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173172516
With Java 8 in GAE standard environment, we can now use standard java thread factory to run the metric reporter in the background in daemon mode, which would not interfere with basic scaling idle timeout as App Engine thread would.
Because the thread is not created by ThreadManager, no App Engine APIs can be called from it. We therefore use GoogleCredential instead of AppIdentityCredential as HttpRequestInitializer, and NetHttpTransport instead of UlrFetchTransport as HttpTransport.
MetricReporter is lazy injected because it depends on jsonCredential retrieved from CloudKms, which is not available in a test environment, causing FrontendServletTest and BackendServletTest to fail.
Some minor re-formatting with google-java-format on edited files.
Lastly removed moe comments in import statement, which makes the linter unhappy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172896227
This originally started as a small change, but quickly grew into a major refactor as I realized the original parameter structure wasn't conducive to a cron task and manual re-runs.
The changes are as follows:
1. Adds DNS metrics to activity reports, thanks to Nick's work with the Zoneman Dremel -> #plx workflow.
2. Surrounds registrar names in transactions reports with quotes, to escape possible commas.
3. Factors out the report generation logic into IcannReportingStager.
4. Assigns default values to the three main parameters
- yearMonth defaults to the previous month
- subdir defaults to "icann/monthly/yearMonth", i.e. "gs://domain-registry-reporting/icann/monthly/yyyy-MM"
- reportType defaults to both reports
5. Adds "Total" row generation logic to transactions reports
- This was a previously overlooked requirement.
6. Adds "MANIFEST.txt" generation and upload logic.
- The MANIFEST lists out which files need to be uploaded in the subdirectory.
7. Increases urlfetch timeout from 5s to 10s in backend tasks.
- Backend tasks should be more latency tolerant anyway, and this reduces the number of incorrect timeouts we see for services like Bigquery which might take some time to respond.
TESTED=Extensive testing in alpha, and ran FOSS test.
TODO: send out an e-mail for report generation and upload, and add reporting to cron.xml
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172738344
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
We're going to need to switch away from Guava's Functions and Predicates for
everything and replace them with the java.util versions. Unfortunately there
does not appear to be an automated tool to do this all at once. Refaster got
close but doesn't seem to care about these particular types of mismatch (I
suspect we're using a different version of the JDK than the outside world;
ours is OK with Guava classes).
This also bumps up Guava to 0.23, which is needed for some new functionality
used in combination with Java 8 features.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170531539
Sometimes requests "die" suddenly, without going through catch/finally blocks.
If this happens, any lock they own will remain locked until it times out (which
can take hours in some cases).
This cl implicitly unlocks any lock if the owner of the lock isn't running
anymore.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168880938
This is needed for the Lock.java enhancement where a lock will be implicitly
released if the request owning it dies.
No matter which solution we want for refactoring the Lock itself, we will need this class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167600314
This is to fix Kokoro, given the directory added in [] Also, added forgotten keyring/api directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598465
We create an injectable LockHandler that just calls the static
Lock.executeWithLocks function.
I'm not sure what's the correct place to put the LockHandler. I think
model/server is only appropriate for the actual datastore lock. This is a "per request" lock, so maybe request/lock?
-----------------------------
This is the initial step in adding the "lock implicitly released on request death" feature, but it's also useful on its own - easier to test Actions when we can use a fake lock.
To keep this CL simple, we keep using the old Lock as is in most places. We just choose a single example to convert to LockHandler to showcase it. Converting all other uses will be in a subsequent CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167357564
The auth logging has been useful, but it now generates a sizeable percentage of all logging, because it spits out three to five lines for every request in the system. This CL reduces that to two to three. We may eventually want to reduce it further, but this is a good start.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164146182
It was not a problem after all to handle multiple scopes. Also added a temp variable to avoid making the same array conversion over and over.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164002903
We want to be safer and more explicit about the authentication needed by the many actions that exist.
As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
The affected actions have been changed to check that the user is logged in by [] so this attribute is no longer needed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159572365
We are going to remove the requireLogin attribute from the action attribute, because it is specific to the UserService API. This is used by four actions:
ConsoleUIAction
RegistrarSettingsAction
RegistrarPaymentSetupAction
RegistrarPaymentAction
Instead, these four actions will now check the login status directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159562335
RdeStagingAction always processed all RDE and BRDA deposits currently outstanding, updating the cursors appropriately and kicking off the upload job. Sometimes we don't want all that. We just want to create a specific deposit by hand, without modifying the cursors or uploading. This CL adds parameters to support that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152415959
The code to authenticate and authorize incoming requests (including via OAuth) has been in the system. This CL actually turns it on, since we are satisfied from logging information that it is not unjustly denying access.
Auth settings are also updated on a few commands missed earlier.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152381820
