Part of the attempt to remove or suppress warnings for deprecated
API use, which will make the Gradle project usable with Intellij.
Currently in the Intellij/Gradle setup, deprecation warnings cause
Intellij build process to fail. Passing -Werror:none flags to javac\
does not have any effect.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=246135737
As it turns out, using Maps::transformValues does not allow us to change the
resulting map--calling Map::put throws an UnsupportedOperationException. As a
result, we have to do this roundabout stream-collect to do a group-by.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=244852373
Collecting by key leads to exceptions if there are multiple client IDs with the
same email address (if we group by client ID in the pipeline). Using
Multimaps::index means that if we're grouping by email, all matches with the
same email get concatenated together
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=242858112
Obviously this is a bad thing and would fail if it ever happened. If this does occur, we will send a warning email.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=240977242
In [] a change would have been made to your project that is incompatible with
your open source integration. To make sure the open source variant of your
project remains working, we have eagerly updated your open source copy to use
Mockito 2. This CL integrates that change into [].
Please read []and
understand the consequences of this change.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=238445356
This CL upgraded google/errorprone plug-in to 2.3.3 and resolved
some warnings detected from the plug-in.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=238047862
The main thrust of this is to create a common POJO that contains email content in a simple way, then have one class that converts that to an email and sends it. Any class that uses email should only have to deal with creating that POJO.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237883643
Icann reports have 3 parameter-provided injections:
- yearMonth
- subdir
- reportType
We move all of them away from the "inner classes" and only @Inject them in the Actions themselves.
This has 2 benefits:
- it's much clearer what all the parameter inputs of the Actions are
- the "inner injected classes" don't assume anything about the Action that uses them - they will work just as well for JSON actions as for "regular" actions.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233625765
We want to make it clear what query (or POST) inputs the user needs to / can give for each Action. That means moving all the @Injects of these parameters to the Actions themselves instead of injecting them in "hidden" indirect dependencies.
This has the extra benefit of allowing these indirect dependencies to work for JSON Actions as well, since the "regular" way we @Inject parameters can corrupt the POST JSON data.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232540758
The daily template is the only one that needs it but we can always pass it in without issue.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=231295089
This eliminates the use of Objectify polymorphism for EPP resources entirely
(yay!), which makes the Registry 3.0 database migration easier.
It is unfortunate that the naming parallelism of EppResources is lost between
ContactResource, HostResource, and DomainResource, but the actual type as far as
Datastore was concerned was DomainBase all along, and it would be a much more
substantial data migration to allow us to continue using the class name
DomainResource now that we're no longer using Objectify polymorphism. This
simply isn't worth it.
This also removes the polymorphic Datastore indexes (which will no longer
function as of this change). The non-polymorphic replacement indexes were added
in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=230930546
For each registrar, the daily email will only include threats that did not appear
in the prior run's email.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228889972
Modularize the code for DNS count reporting to allow it to be customized for
more flexible systems.
Tested:
Uploaded to alpha with hacks to allow admin initiating and logging from the
DnsCountQueryCoordinatorModule, verified that the provider function is invoked and
that the action runs successfully.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225225587
Eventually the Publish action will control daily/monthly sending and provide
the correct threats to email. The goal of this PR is to entirely separate
the "sending email" functionality from the "parsing threat matches"
functionality.
The PublishAction will figure out if the monthly emails should be sent out,
then will ask the Spec11ThreatMatchesParser for the monthly threats (if
appropriate) and the new threat matches for today. It will then pass those
matches and the appropriate email subject+body to the email utils class,
whose only job is to format and send the emails.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224869643
This is for consistency, mostly the LocalDate fields added in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224525451
Add a sendSpec11Email parameter that allows us to only send the email on
one run per month. Next, we will compute the diffs between the daily runs
and send daily emails with those diffs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=224404653
This is surprisingly difficult to fix correctly, so for now, just don't
worry about the content type.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221694745
This turns on spec11 reporting in production by adding it to the cron.xml, generating the report and sending an e-mail with a list of all problematic registrations to the associated registrar on the 2nd of each month at 15:00Z (11am EST)
This also tweaks the e-mail template a bit according to suggestions from Bruno.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=213031440
This adds the terminal step of the Spec11 pipeline- processing the output of
the Beam pipeline to send an e-mail to each registrar informing them of
identified 'bad urls.'
This also factors out methods common between invoicing (which uses similar beam pipeline tools) and spec11 to the common superpackage ReportingModule + ReportingUtils classes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=210932496
This adds actual subdomain verification via the SafeBrowsing API to the Spec11
pipeline, as well as on-the-fly KMS decryption via the GenerateSpec11Action to
securely store our API key in source code.
Testing the interaction becomes difficult due to serialization requirements, and will be significantly expanded in the next cl. For now, it verifies basic end-to-end pipeline behavior.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208092942
This removes the following unnecessary imports:
//third_party/java_src/gtld/java/google/registry/bigquery
//third_party/java_src/gtld/java/google/registry/config
//third_party/java_src/gtld/java/google/registry/cron
//third_party/java_src/gtld/java/google/registry/dns
//third_party/java_src/gtld/java/google/registry/gcs
//third_party/java_src/gtld/java/google/registry/mapreduce
//third_party/java_src/gtld/java/google/registry/model
//third_party/java_src/gtld/java/google/registry/module/backend
//third_party/java_src/gtld/java/google/registry/module/frontend
//third_party/java_src/gtld/java/google/registry/module/pubapi
//third_party/java_src/gtld/java/google/registry/module/tools
//third_party/java_src/gtld/java/google/registry/request
//third_party/java_src/gtld/java/google/registry/security
//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers
//third_party/java_src/gtld/java/google/registry/util
//third_party/java_src/gtld/java/google/registry/xjc
//third_party/java_src/gtld/javatests/google/registry/model
//third_party/java_src/gtld/javatests/google/registry/testing
//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java_src/gtld/java/google/registry/bigquery,//third_party/java_src/gtld/java/google/registry/config,//third_party/java_src/gtld/java/google/registry/cron,//third_party/java_src/gtld/java/google/registry/dns,//third_party/java_src/gtld/java/google/registry/gcs,//third_party/java_src/gtld/java/google/registry/mapreduce,//third_party/java_src/gtld/java/google/registry/model,//third_party/java_src/gtld/java/google/registry/module/backend,//third_party/java_src/gtld/java/google/registry/module/frontend,//third_party/java_src/gtld/java/google/registry/module/pubapi,//third_party/java_src/gtld/java/google/registry/module/tools,//third_party/java_src/gtld/java/google/registry/request,//third_party/java_src/gtld/java/google/registry/security,//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers,//third_party/java_src/gtld/java/google/registry/util,//third_party/java_src/gtld/java/google/registry/xjc,//third_party/java_src/gtld/javatests/google/registry/model,//third_party/java_src/gtld/javatests/google/registry/testing,//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202652421
This removes the following unnecessary imports:
//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
This removes the following unnecessary imports:
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='[],[],[],[],[],[],[],[],[],[],[],[],[],[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202325520
This affects JSR305, JSR330, and Guava annotations.
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
Now that the large zone re-signing test is complete, we no longer need it.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199507075
Explicit transfer acks/nacks reverse the roles for transaction reporting
tabulation- this adds a quick check to account for this going forward.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=199474444
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.
No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.
Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.
For more information, see []
Base CL: 197826149
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198560170
This adds a parameter to control invoice reporting, which defaults to false for
now (since we plan on manually adjusting the invoice next month).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193708323
This moves the default yearMonth logic into a common ReportingModule, rather than the coarse-scoped BackendModule, which may not want the default parameter extraction logic, as well as moving the 'yearMonth' parameter constant to the common package it's used in. This also provides a basis for future consolidation of the ReportingEmailUtils and BillingEmailUtils classes, which have modest overlap.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=183130311
This makes a few cosmetic changes that prepares the pipeline for production.
Namely:
- Converts file names to include the input yearMonth, mostly mirroring the original invoicing pipeline.
- Factors out the yearMonth logic from the reporting module to the more common backend module. We will likely use the default yearMonth logic in other backend tasks (such as spec11 reporting).
- Adds the "withTemplateCompatability" flag to the Bigquery read, which allows multiple uses of the same template.
- Adds the 'billing' task queue, which retries up to 5 times every 3 minutes, which is about the rate we desire for checking if the pipeline is complete.
- Adds a shell 'invoicing upload' class, which tests the retry semantics we want for post-generation work (e-mailing the invoice to crr-tech, and publishing detail reports)
While this cl may look big, it's mostly just a refactor and setting up boilerplate needed to frame the upload logic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=179849586
This is in preparation for running the automatic refactoring script that
will replace all ExpectedExceptions with use of JUnit 4.13's assertThrows/
expectThrows.
Note that I have recorded the callsites of assertions about EppExceptions
being marshallable and will edit those specific assertions back in after
running the automatic refactoring script (which do not understand these).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178812403
The only remaining methods on ExceptionRule after this are methods that
also exist on ExpectedException, which will allow us to, in the next CL,
swap out the one for the other and then run the automated refactoring to
turn it all into assertThrows/expectThrows.
Note that there were some assertions about root causes that couldn't
easily be turned into ExpectedException invocations, so I simply
converted them directly to usages of assertThrows/expectThrows.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=178623431
The scheme is:
- loadBytes: returns a ByteSource of the data
- loadFile: returns a string using UTF8 encoding, optionally applying
substitutions
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177606406
There is a big mix of different "load Resources" from different libraries
depending on where you were and what type of resource you want. Now there is a
clear hirarchy:
ResourceUtils:
for use in actual (non-test) code, reads a file from a context directory
TestDataHelper (uses ResourceUtils):
for use in tests, reads a file from a context directory + "/testdata". Also
caches the resource so calling it multiple times with the same file will
not read the file multiple times.
Library specific helpers (e.g. ToolsTestData) (uses TestDataHelper):
for use in that library's tests only, reads from a specific testdata directory.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177027533
This converts the upload task from a cron job to a task chained after staging.
This ensures the upload job only occurs when its dependencies are met, and
provides a faster turnaround time to verify both the staging and upload jobs
are complete.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=175045489
This is a final refactor to address Nick's comments in [] where YearMonth really should be injected as a Joda type instead of a raw string, and the HTTP parameters should be separate from the default-provided dependencies.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173539965
This change:
- Adds retries to the staging action
- Emails domain-registry-eng@ upon completion of either action
- Simplifies logging to be more useful
TODO: fix up Module @Inject naming conventions and yearMonth injection
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173294822
