This moves the default yearMonth logic into a common ReportingModule, rather than the coarse-scoped BackendModule, which may not want the default parameter extraction logic, as well as moving the 'yearMonth' parameter constant to the common package it's used in. This also provides a basis for future consolidation of the ReportingEmailUtils and BillingEmailUtils classes, which have modest overlap.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=183130311
NotModifiedException was using HttpServletResponse.SC_NOT_FOUND instead of SC_NOT_MODIFIED (likely an autocomplete typo).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182976671
This fixes up the following problems:
1. Using string concatenation instead of the formatting variant methods.
2. Logging or swallowing exception messages without logging the exception
itself (this swallows the stack trace).
3. Unnecessary logging on re-thrown exceptions.
4. Unnecessary use of formatting variant methods when not necessary.
5. Complicated logging statements involving significant processing not being
wrapped inside of a logging level check.
6. Redundant logging both of an exception itself and its message (this is
unnecessary duplication).
7. Use of the base Logger class instead of our FormattingLogger class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=182419837
This removes some qualifiers that aren't necessary (e.g. public/abstract on interfaces, private on enum constructors, final on private methods, static on nested interfaces/enums), uses Java 8 lambdas and features where that's an improvement
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177182945
Most of the time, we don't expect incoming requests to have an authorization
header. So this statement gets printed a lot, and doesn't provide much useful
information. We already have a statement listing what type of
authentication/authorization is required by the endpoint, and other statements
indicating either that authorization was successful with a particular method or
was not successful at all.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175969652
The former method -- a config string -- was cumbersome, as each Nomulus system would have to configure the link base to its own URL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173567021
We were relying on Dagger to validate the IP address, but that resulted in 500 errors when the IP address was not valid, which is undesirable. Instead, accept the parameters as strings, then convert them to IP addresses and throw a proper error when conversion fails.
Also fixes an improperly specified test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173172516
With Java 8 in GAE standard environment, we can now use standard java thread factory to run the metric reporter in the background in daemon mode, which would not interfere with basic scaling idle timeout as App Engine thread would.
Because the thread is not created by ThreadManager, no App Engine APIs can be called from it. We therefore use GoogleCredential instead of AppIdentityCredential as HttpRequestInitializer, and NetHttpTransport instead of UlrFetchTransport as HttpTransport.
MetricReporter is lazy injected because it depends on jsonCredential retrieved from CloudKms, which is not available in a test environment, causing FrontendServletTest and BackendServletTest to fail.
Some minor re-formatting with google-java-format on edited files.
Lastly removed moe comments in import statement, which makes the linter unhappy.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172896227
This originally started as a small change, but quickly grew into a major refactor as I realized the original parameter structure wasn't conducive to a cron task and manual re-runs.
The changes are as follows:
1. Adds DNS metrics to activity reports, thanks to Nick's work with the Zoneman Dremel -> #plx workflow.
2. Surrounds registrar names in transactions reports with quotes, to escape possible commas.
3. Factors out the report generation logic into IcannReportingStager.
4. Assigns default values to the three main parameters
- yearMonth defaults to the previous month
- subdir defaults to "icann/monthly/yearMonth", i.e. "gs://domain-registry-reporting/icann/monthly/yyyy-MM"
- reportType defaults to both reports
5. Adds "Total" row generation logic to transactions reports
- This was a previously overlooked requirement.
6. Adds "MANIFEST.txt" generation and upload logic.
- The MANIFEST lists out which files need to be uploaded in the subdirectory.
7. Increases urlfetch timeout from 5s to 10s in backend tasks.
- Backend tasks should be more latency tolerant anyway, and this reduces the number of incorrect timeouts we see for services like Bigquery which might take some time to respond.
TESTED=Extensive testing in alpha, and ran FOSS test.
TODO: send out an e-mail for report generation and upload, and add reporting to cron.xml
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172738344
This was a surprisingly involved change. Some of the difficulties included
java.util.Optional purposely not being Serializable (so I had to move a
few Optionals in mapreduce classes to @Nullable) and having to add the Truth
Java8 extension library for assertion support.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171863777
We're going to need to switch away from Guava's Functions and Predicates for
everything and replace them with the java.util versions. Unfortunately there
does not appear to be an automated tool to do this all at once. Refaster got
close but doesn't seem to care about these particular types of mismatch (I
suspect we're using a different version of the JDK than the outside world;
ours is OK with Guava classes).
This also bumps up Guava to 0.23, which is needed for some new functionality
used in combination with Java 8 features.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170531539
Sometimes requests "die" suddenly, without going through catch/finally blocks.
If this happens, any lock they own will remain locked until it times out (which
can take hours in some cases).
This cl implicitly unlocks any lock if the owner of the lock isn't running
anymore.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168880938
This is needed for the Lock.java enhancement where a lock will be implicitly
released if the request owning it dies.
No matter which solution we want for refactoring the Lock itself, we will need this class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167600314
This is to fix Kokoro, given the directory added in [] Also, added forgotten keyring/api directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167598465
We create an injectable LockHandler that just calls the static
Lock.executeWithLocks function.
I'm not sure what's the correct place to put the LockHandler. I think
model/server is only appropriate for the actual datastore lock. This is a "per request" lock, so maybe request/lock?
-----------------------------
This is the initial step in adding the "lock implicitly released on request death" feature, but it's also useful on its own - easier to test Actions when we can use a fake lock.
To keep this CL simple, we keep using the old Lock as is in most places. We just choose a single example to convert to LockHandler to showcase it. Converting all other uses will be in a subsequent CL.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167357564
The auth logging has been useful, but it now generates a sizeable percentage of all logging, because it spits out three to five lines for every request in the system. This CL reduces that to two to three. We may eventually want to reduce it further, but this is a good start.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164146182
It was not a problem after all to handle multiple scopes. Also added a temp variable to avoid making the same array conversion over and over.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164002903
We want to be safer and more explicit about the authentication needed by the many actions that exist.
As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
The affected actions have been changed to check that the user is logged in by [] so this attribute is no longer needed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159572365
We are going to remove the requireLogin attribute from the action attribute, because it is specific to the UserService API. This is used by four actions:
ConsoleUIAction
RegistrarSettingsAction
RegistrarPaymentSetupAction
RegistrarPaymentAction
Instead, these four actions will now check the login status directly.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159562335
RdeStagingAction always processed all RDE and BRDA deposits currently outstanding, updating the cursors appropriately and kicking off the upload job. Sometimes we don't want all that. We just want to create a specific deposit by hand, without modifying the cursors or uploading. This CL adds parameters to support that.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152415959
The code to authenticate and authorize incoming requests (including via OAuth) has been in the system. This CL actually turns it on, since we are satisfied from logging information that it is not unjustly denying access.
Auth settings are also updated on a few commands missed earlier.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=152381820
A test has been added to RequestHandlerTest, making sure that, while we merely log errors for the time being, the correct dummy AuthResult is being created.
Most actions use the default settings, which have been changed to INTERNAL / APP / IGNORED. Actions with non-default settings are:
INTERNAL/NONE/PUBLIC (non-auth public endpoints)
CheckApiAction
WhoisHttpServer
Rdap*Action
INTERNAL,API/APP/ADMIN (things currently protected by web.xml)
EppTlsAction
EppToolAction
CreateGroupsAction
CreatePremiumListAction
DeleteEntityAction
List*sAction
UpdatePremiumListAction
VerifyOteAction
WhoisServer
INTERNAL,API,LEGACY/USER/PUBLIC (registrar console)
RegistrarPaymentAction
RegistrarPaymentSetupAction
RegistrarSettingsAction
EppConsoleAction
INTERNAL,API,LEGACY/NONE/PUBLIC (registrar console main page)
ConsoleUiAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149761652
[] changed RequestHandler to log a warning instead of returning an error to the user when authentication failed. Unfortunately, it didn't handle the resulting absent value of AuthResult. I don't yet know why the tests pass, and plan to add a test to check for this situation, but I wanted to start by fixing the problem as soon as possible, as it seems like this would result in exceptions any time a request was handled which didn't pass authentication.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149136265
This is the first step in rolling out the changes so that we can check via logging whether turning on the logic would reject anything it should not.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=149050878
This follows up on Brian's work to transition not just to a new format
with an empty scope value, but instead to replace the existing format
entirely with a new one that:
1) includes a version number to support future format migrations
2) doesn't include a field for the scope at all, since scoping the
tokens adds no real security benefit and just makes verification
more difficult
3) replaces the raw SHA-256 hash with a SHA-256 HMAC instead, as a
best practice to avoid length-extension attacks [1], even though
in our particular case they would only be able to extend the
timestamp and would thus be relatively innocuous
The new format will be produced by calling generateToken(), and the
scope-accepting version is renamed to generateLegacyToken() in addition
to its existing deprecation, for maximum clarity.
I changed the validateToken() logic to stop accepting a scope entirely;
when validating a legacy-style token, we'll test it against the two
existing legacy scope values ("admin" and "console") and accept it if
it matches either one.
Note that this means the xsrfScope parameter in @Action is now wholly
obsolete; I'll remove it in a follow-up to avoid bringing extra files
into this CL.
After this CL hits production, the next one will replace all calls to
generateLegacyToken() with generateToken(). Once that CL is deployed,
the last step will be removing the legacy fallback in validateToken().
[1] See https://en.wikipedia.org/wiki/Length_extension_attack
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148936805
This provides a safeguard against using TypeInstantiator to resolve the component class, where if resolution is done incorrectly, you end up with java.lang.Object. Formerly, that would have "succeeded" in generating a Router for Object, which of course has no methods that return @Action classes. Such a router is pretty useless, so it's better to make Router stricter and have it fail if you give it such a class by accident.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148353224
It turns out this type parameter was never necessary. A builder only needs the reflexive second type parameter when you want to have a builder inheritance hierarchy where the descendant builders have methods that the ancestor builder doesn't. In that case, the type param enables the ancestor builder's setter methods to automatically return the correct derived type, so that if you start with a derived builder, you can call a setter method inherited from an ancestor and then continue the chain with setters from the derived builder (e.g. new ContactResource.Builder().setCreationTime(now).setContactId(), which otherwise would have returned an EppResource.Builder from setCreationTime(), at which point the call to setContactId() would not compile).
Even then, it's not strictly necessary to use the type parameter, since you could instead just have each derived type override every inherited method to specify itself as the return type. But that would be a lot of extra boilerplate and brittleness.
Anyway, in this case, there is a builder hierarchy, but RequestComponentBuilder specifies all the methods that we're ever going to want on our builders, so there's never any need to be able to call specific derived builder methods. We only even need the individual builder classes so that Dagger can generate them separately for each component.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148269178
The one-day validity period is also moved from the caller into XsrfTokenManager.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=147857716
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072
We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018
A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55
The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.
Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:
- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
