Commit graph

2714 commits

Author SHA1 Message Date
shicong
66d98c8d66 Add SetNumInstancesCommand in Nomulus tool to adjust the number of instances
for a given service and version at runtime.

Note that this CL only supports the adjustment for a given service and version. I will add another functionality to let this command be able to detect all non-live versions automatically and apply the adjustment.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221092001
2018-11-12 14:51:40 -05:00
guyben
557984bb75 Add support G-Suite group whose members have ADMIN access to registrar console
After this CL, "support" accounts (accounts that are part of the "support" G-Suite group) will the same access to the registrar console as GCP "admins". However, they don't won't have access to the GCP project itself.

We could give them their own Role in the future (say SUPPORT) and give them different access than "admins", but right now we don't need it and YAGNI or something :)

NOTE: we identify users by their email (they need to be logged in to a google account). I don't know if that's best practice, since I guess different google accounts might have the same email address. However, G-Suite groups' membership is by email so there's not much we can do about it if we want to use G-Suite groups.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220804273
2018-11-12 14:51:40 -05:00
weiminyu
783c010ab4 Add a IDE helper plugin for annotation processors
Without the net.ltgt.apt plugin, Dagger does not retain generated source code,
and AutoValue mixes generated source code with classes. The new plugin fixes this
problem, making it easier to work with IDEs.

Based on limit experience, Intellij Idea can properly import this project. We
may consider adding IDE-specific plugins in the future.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220802052
2018-11-12 14:51:40 -05:00
mmuller
9312cb3936 Don't use gradlew from travis
Since we're explicitly downloading a version of travis anyway, just run that
and don't bother with creating gradlew which does nothing but download the
same version over again.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220618139
2018-11-12 14:51:40 -05:00
guyben
61a5cf307e Add "Admin" tab to the registrar console
This tab will set the "allowedTlds", but might have other functionality in the
future.

It is based on (branches from) the security-settings tab, because I'm copying the functionality of the "whitelisted IPs" to the "allowed TLDs": they are both lists of "arbitrary" strings that you can remove from and add to.

There are a lot of moving parts in this CL, because of how all the different elements need to interact, and how intertwined they are (for example, we need to disable the admin-settings view for non admins both in the soy and in the JS code)

It's really time to refactor the console given all we've learned... :/

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220373443
2018-11-12 14:51:40 -05:00
jianglai
9b10c116f3 Do not create a logger during initialization in CidrAddressBlock
This is patched from [] We should have done this when we migrated to
Flogger.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219860046
2018-11-12 14:51:40 -05:00
guyben
9ce07db38a Make OutputEncapsulator a CommandRunner
This is in preparation for having other "command changing things" like
redirecting to file and maybe variable substitutions in the arguments.

"On the way" added a
RUNNING "some_command" "--some_flag" "some_value"
to the output encapsulator so that if we run multiple commands, we know what
command was called where.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219837452
2018-11-12 14:51:40 -05:00
Michael Muller
9b4d175471 Testing a direct "gradle build" (no gradlew) 2018-11-07 11:15:52 -05:00
mcilwain
f59005ad35 Fix sender email address for invoicing alerts
It was failing to send alert emails because the email address it was
constructing did not have permission through GAE to send emails. This switches
it over to using the send from email address already in use elsewhere in the app
that does successfully send emails.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219812019
2018-11-02 14:50:52 -04:00
weiminyu
6a870e5820 Check for unused dependencies
Enabled unused-dependency check using nebula-lint.

Dependencies that are not used by compile or testing are
labeled with 'maybe_runtime". We leave these dependencies
in the script for easy reference. Before launching Gradle-based
release process we must determine which of these should
be removed and which should be relabeled as runtime.

Label assignment:
- All dependencies recommended for removal from 'compile'
  are changed to maybe_runtime
- All dependencies recommended for move from 'compile' to
  testCompile are split into two lines, one with testCompile,
  the other maybe_runtime

Incidentally, Gradle 4.10.2 needs a groovy upgrade before
it can work with Oracle JDK 11.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219803797
2018-11-02 14:49:18 -04:00
jianglai
b5856a1467 Fix FOSS build
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219794867
2018-11-02 14:45:59 -04:00
guyben
3f6585fccc Reduce duplicate code in the servlets
Currently, all 4 servlets (backend, frontend, pubapi, tools) have duplicates of
the same exact code.

That's an anti-pattern!

Created a ServletBase they can all extend which has the duplicate code.

As a bonus, the tools servlet now runs the metric reporter, meaning tool
related metrics will now be reported!

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219792176
2018-11-02 14:44:22 -04:00
mmuller
545b68ad9a Add a .travis.yml file
Add the Travis-CI configuration file, which will allow our Gradle build to be
tested from Github with travis.

TESTED:
  1) Repeatedly tweaked this on own local fork of google/nomulus until I was able to successfully build.
  2) Verified that this shows up in the tarball after "blaze build :opensource-staging"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219678030
2018-11-02 14:42:45 -04:00
guyben
08290e6b87 Fix the tool that was broken in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219629901
2018-11-02 14:39:36 -04:00
Michael Muller
79bbd5e29d Upped the build time to 45 minutes. 2018-11-01 08:48:07 -04:00
weiminyu
00d0284d5b Add nebula-lint plugin to Gradle scripts
Updated the project config so that the plugin can work.

Reverted to the deprecated compile/testCompile labels
so that the plugin can work. This plugin provides valuable
checks that we do not want to give up.

Added undeclared-dependency check. Still need to add
unused-dependency and one-version check.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219406600
2018-10-31 19:15:03 -04:00
mountford
09202562c7 RDAP: Add link to static TOS page
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219332975
2018-10-31 19:13:23 -04:00
guyben
57f06258d3 Add metrics measuring all request processing times
The cardinality of this new metric is:
buckets - 16
path + method - around 100 (the number of Actions we have)
authLevel - at most 3
success - 2

Total: 16*100*3*2 = 9,600

This is still low, especially for the value it could give in understanding our system (graphs of all endpoints, how often are they called, how long they take, how often do they fail)

Instead of "success true/false", we might want to give the actual status code. This can be a bit annoying because HttpServletResponse doesn't have a getStatus.

But it's possible, and worth considering.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219312400
2018-10-31 19:11:39 -04:00
guyben
a45d3d3bc7 Add a log statement at the very end of our code
Having a log at the very begining of "our" code helped us find issues with
App-Engine's dispatcher, where we could clearly see in the logs that "our" code
started many seconds after the request came in.

We now suspect there is something similar going on after the end of "our" code
- where the reply is sent back many seconds after our code finished running.

To make sure - we add a log statement at the very last line of "our" code, so
we know exactly when it ended.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219309962
2018-10-31 19:09:51 -04:00
Michael Muller
c1d586a211 Make travis wait up to 30 minutes on build without output.
Also add some comments to .travis.yml.
2018-10-31 17:13:16 -04:00
Michael Muller
beae72f6ba Set TERM=dumb in environment to limit build output
The status information gradle prints out doesn't look very good in Travis.  In
fact, it causes the logfile to exceed the maximum allowed by Travis.
2018-10-31 16:44:26 -04:00
Michael Muller
a3a474964c Do "gradlew build" insetead of just gradlew. 2018-10-31 16:00:18 -04:00
Michael Muller
1b543d55a5 Fixed url and unzip in gradle download. 2018-10-31 15:57:03 -04:00
Michael Muller
f5408b36e3 Attempt to install gradle on travis.
Try installing gradle instead of calling gradlew, which it turns out requires
its jarfiles to be presesnt in order to work.
2018-10-31 15:39:28 -04:00
Michael Muller
a4d10f03d6 Use checked-in "gradlew" instead of rebuilding it.
We want to use a specific version of gradle, not the 4.0.1 version that travis
has lying around.
2018-10-31 15:10:13 -04:00
Michael Muller
f16d8019e5 Checking in gradlew/gradlew.bat. 2018-10-31 15:07:57 -04:00
Michael Muller
0950333434 Changed .travis.yml script to attempt gradle build. 2018-10-31 14:57:43 -04:00
Michael Muller
b381b02add Resolved merge conflict 2018-10-31 14:41:05 -04:00
mmuller
a76300f76c Do output encapsulation in a try/with
Move the shell output encapsulation so that we don't double-wrap on a
premature exit.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219136896
2018-10-29 15:42:41 -04:00
jianglai
c375b0d5f4 Add javax.activation as a dependency of jaxb
In Java 9+, this module is no longer loaded by default. JAXB needs it to compile.

Also changed the root project name to "nomulus".

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219000053
2018-10-29 15:41:06 -04:00
shicong
21f706bf24 Generate client id menu dynamically instead of hard coding the mapping from
role to client id.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218913784
2018-10-29 15:37:32 -04:00
guyben
b48061b792 Refactor AppEngineConnection
AppEngineConnection can now connect to all services and not just the tools.

The default is still the tools.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218734983
2018-10-29 15:34:12 -04:00
guyben
97aa98eb35 Add metrics for registrar console requests
Cardinality of this metric:

clientId: there are currently 650 (on sandbox, because of OTE), and 200 on production.
explicitClientId: 2
roles: 2 now, might be 3 soon if we add vendors
status: 2

So we're talking about a cardinality of 2,000-8,000. Less when you consider that registrars only seldom actually need to access the console (certainly not daily or even weekly).

Compare with, e.g., the /epp/processing_time from the above EppMetrics.java which has:
Epp commands: 26 (manual counting)
client IDs: 200 on prod
status: the actual status CODE of the command. Can have many values, but looking at the past few weeks' metrics I counted 20
Note that not every command results in every status. Looking a few weeks back we can see around 80-100 (commands+status) combination.
buckets: 16

so that's over 250,000-1,000,000 cardinality, on a very high-volume metric.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218699280
2018-10-25 14:51:58 -04:00
weiminyu
f6d9b46622 Reorganize Gradle dependencies
Move configs and dependencies from root script to subprojects.
There is not too much commonality between subprojects. Besides,
the gradle lint plugin (not added to this project yet) does not
handle inherited dependencies properly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218698818
2018-10-25 14:50:26 -04:00
jianglai
ca97d66dd7 Update .gitignore for gradle related files
Ignore all gradle build artifacts.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218693364
2018-10-25 14:47:10 -04:00
jianglai
f0195f3ed6 Explicitly state which Gradle version is tested.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218687425
2018-10-25 14:45:30 -04:00
jianglai
85d971c943 Allow admin to set AllowedTlds in RegistrarSettingsAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218508076
2018-10-25 14:43:54 -04:00
weiminyu
6f155ed3d2 Properly label compile-only dependencies in Gradle script
Dependencies labeled as 'neverlink' in bazel should have two
labels in Gradle: compileOnly and testCompileOnly/testImplementation.
This excludes the dependency from deployment archive.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218394354
2018-10-25 14:42:17 -04:00
jianglai
57d95d4bec Refactor TmchCertificateAuthority
The main goal of this CL is to make the clock constructor injectable (so that tests do not need to use InjectRule to override the clock). The complication is that the clock is used by two static caches (ROOT_CACHE directly and CRL_CACHE indirectly). The clock is not actually used to construct the lock, but rather to verify that the root certificate is within its validity period.

For ROOT_CACHE we move the verification to its call sites. This adds a bit overhead because the validity check happens every time the cache is called, not just when the cache is built or refreshed. However this check is rather cheap.  Also the resources are included in the jar and the cache is valid for 1 year. Given that we deploy every week, there's not much point making it an expiring cache rather than a static map.

For CRL_CACHE we change the key to a tuple of TmchCaMode and X509Certificate. The certificate is no longer provided from the ROOT_CACHE directly and must be verified before it is provided as a cache key. We left the CRL verification inside the cache loader because it (signature verification) is more expensive compared to simple expiration check, and we do not want to do this every time the cache is called.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218385684
2018-10-25 14:40:38 -04:00
jianglai
589e98a1db Explicitly set the shard size when saving the claims list
This allows us to get rid of the use of InjectRule in tests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218221409
2018-10-22 19:13:11 -04:00
jianglai
6a560c18ce Do not monitor gradle temp files changes
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218213950
2018-10-22 19:11:39 -04:00
jianglai
e39cc04110 Test AsyncFlowMetrics without mock
Instead of verifying interactions on the mocks, we instead assert on the real test subject directly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218209556
2018-10-22 19:10:01 -04:00
guyben
d2ca67460c Allow admins read/write access to all registrar in web console
This CL removes the "READ vs UPDATE" feature completely. Now anyone with access
has full read+write access.

We still keep track of which role a user has (did they get access "explicitly"
because they are an "allowed access" contact? Or do they have access because
they are admins?) for the logs and UI, and also so we could in the (very near)
future have features only available to admins.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=218169608
2018-10-22 19:08:09 -04:00
jianglai
2020dcb50f Refactor StringGenerator bindings
Make every dependency request explicit on what encoding is used. Also get rid of InjectRule in XjcToDomainResourceConverterTest.

Random number generator providers are separated to secure and insecure ones. The insecure ones must be explicitly requested (usually for use cases where security is not of concern, for better speed).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217921422
2018-10-22 19:06:35 -04:00
guyben
1a4aae8f7d Use RDAP test tools in RdapActionBaseTest
This should have been part of []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217881035
2018-10-22 19:03:14 -04:00
guyben
3a3b0b738a Centralize RDAP test initialization and login
All RDAP actions inherit from a common RdapActionBase class.

All RDAP search actions inherit from a comman RdapSearchActionBase class, which inherits from RdapActionBase

Each of the base classes has @Before initialization needed for the tests, as well as utility functions (such as login and logout).

Currently, these were copied in all test classes. Instead, we created a similar test inheritance tree to centralize the initialization and place common utility functions.

This way, the @Before of every test only needs to initialize the variables new to the specific action, making the code somewhat clearer.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217856844
2018-10-22 19:01:37 -04:00
jianglai
b254269d2f Refactor LordnTask to LordnTaskUtils
Made it clear that it is a util class and moved some of the functions only used in NordnUploadAction (to NordnUploadAction). Also used Retrier to handle retries when leasing tasks.

These changes allow us to no longer use InjectRule in related unit tests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217761117
2018-10-22 18:59:48 -04:00
guyben
0f1f418034 Add registrar contact to OTE registrars
When creating the various registrar objects in Sandbox for OTE, we also give access to all the registrars' data to a given google account (identified by the email)

This email has to belong to the registry's G-Suite account, just like in the registrar_contact command.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217728407
2018-10-22 18:58:06 -04:00
jianglai
bc2c0350d7 Remove InjectRule in UrlFetchUtilsTest
It should have been removed in []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217728371
2018-10-22 18:56:30 -04:00
jianglai
3daf62a486 Remove the use of InjectRule in IdnLabelValidatorTest
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=217707263
2018-10-22 18:54:53 -04:00