Commit graph

2590 commits

Author SHA1 Message Date
jianglai
4598c5f105 Remove unnecessary Dagger qualifier
The @JsonPayload qualifier is not used because the field are contrustor injected.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222829281
2018-12-03 18:59:40 -05:00
shicong
d20b83c820 Add option --non_live_versions to set_num_instances command
This commit introduced a new flag to enable SetNumInstancesCommand to
be able to set the number of instances for all non-live versions for
a given service or for all deployed services.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222826003
2018-12-03 18:58:05 -05:00
guyben
19b7a7b3ec Allow only OWNERs to change owner-related data on registrar console
The console will have 2 different "updatable things":
- only ADMINs (GAE-admins and users in the support G-Suite group) can change the things in the "admin settings" tab (currently just the allowed TLDs)
- only OWNERs can change things from the other tabs: WHOIS info, certificates, whitelisted IPs, contacts etc.

Also, all ADMINs are now OWNERS of "non-REAL" registrars. Meaning - we're only
preventing ADMINs from editing "REAL" registrars (usually in production).

Specifically, OTE registrars on sandbox are NOT "REAL", meaning ADMINS will
still be able to update them.

This only changes the backend (registrar-settings endpoint). As-is, the console
website will still make ADMINs *think* they can change everything, but if they
try - they will get an error.

Changing the frontend will happen in the next CL - because I want to get this
out this release cycle and getting JS reviewed takes a long time :(

TESTED=deployed to alpha, and saw I can't update fields even as admin on REAL
registrars, but could change it on non-REAL registrars. Also checked that I can
update the allowed TLDs on REAL registrars

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222698270
2018-12-03 18:56:28 -05:00
guyben
5f283ebd09 Use AuthenticatedRegistrarAccessor in EppConsoleAction
EppConsoleAction still "manually" checks access by going over the
RegistrarContacts. We need it to use AuthenticatedRegistrarAccessor just like
every other part of the registrar console.

We still need to remove the (now unneeded) login EPP sent by the console, but that's left for a followup CL.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222404208
2018-12-03 18:51:40 -05:00
jianglai
eca3461dec Add a build badge for gradle
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222315861
2018-12-03 18:50:05 -05:00
jianglai
a0aeec6f2f Update bazel version to 0.19.2
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222309479
2018-12-03 18:48:28 -05:00
jianglai
87f30f0870 Add .gcloudignore and cloudbuild.yaml to open source build
Forgot to do this when I exported... Will have to do another export after this CL is added.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222296851
2018-11-20 16:41:36 -05:00
guyben
274b7115d4 Block ability to remove allowed TLDs from the registrar console
This is a temporary measure until we implement access control for Support.

Once we implement access control, we will only block Support from removing TLDs
on production.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222180321
2018-11-20 16:03:06 -05:00
jianglai
f46f817f9e Create proxy Docker image with Gradle
This CL adds two ways to build an docker image with gradle:

1) Adds a :proxy:deployJar task that builds an uber jar that contains all runtime dependencies. The jar can then be add to a docker image by calling docker with the added Dockerfile. The base image for this image can be both distroless java or openjdk:alpine.

2) Uses the Gradle distribution plugin to build a distribution tar file that contains all dependencies (as separate jar files) and a run script that sets up the classpath before calling the main class. Then the docker application plugin can build a docker image (with the dockerBuildImage task) using the application tar file. This only works with openjdk:alpline base image as the distroless java image does not contain a shell and therefore the script created by the distribution plugin cannot be launched.

We may later decide to use one of the method and remove the other.

Also adds an outcast test pattern that caused the tests to be flaky.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222145192
2018-11-19 18:19:27 -05:00
jianglai
118367dc6b Move proxy to its separate project
It can be run directly by "gradle :proxy:run"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222102849
2018-11-19 18:17:47 -05:00
jianglai
997b49f7ca Move util to its separate project
It will make it easier later to have the proxy project depend on it, rather
than on the entire core project

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222081994
2018-11-19 18:16:03 -05:00
jianglai
f7e95e1ff1 Refactor gradle build script
1. Updated nebula lint version to 10.3.1
2. Do not apply plugins in the root projects that are not needed.
3. Only do linting when a build is successful, so that build failure message are not flushed by linter warnings.
4. Added explicitly stated java source and target version.
5. Moved source sets set up to the root project so that it applies to all subprojects. Currently there is only "core", but we will add at least "proxy" and "util" later, both of which will use mostly the same source sets (but with additional inclusion rules to only build classes in a specific sub folder in the source tree). By putting the set up closure inside the subprojects block in the root script, it can be reused.
6. Rename maybe_runtime configuration to maybeRuntime, which is consistent with other camelCase configuration names like testCompile.
7. Added a runtime dependency of the JAXB API which is removed from Java SE as of version 11.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222081092
2018-11-19 18:14:09 -05:00
jianglai
02a23c6ca6 Remove unnecessary source sets
These source sets are not configured correctly (no classpaths for example) and
they subsequently cannot compile. The special tests are put into separate tasks that
run from the same source set (test).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=222080273
2018-11-19 18:12:12 -05:00
weiminyu
676fa422db Use proper external dependency references recognized by moe
This fixes kokoro build failure.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221830188
2018-11-16 17:03:50 -05:00
weiminyu
af21b0c32b Add a package to open source
Add {java,tests}/google/registry/export/datastore to open source.
This is part of the migration to Datastore Managed Import/Export
for backup.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221800709
2018-11-16 17:02:15 -05:00
shicong
36c6265980 Make BillingModule.OVERALL_INVOICE_PREFIX configurable
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221700223
2018-11-16 17:00:38 -05:00
mcilwain
45f63cbecf Fix failing email sending FOSS test
This is surprisingly difficult to fix correctly, so for now, just don't
worry about the content type.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221694745
2018-11-16 16:59:05 -05:00
jianglai
86007622f7 Remove proxy's dependency on config
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221666668
2018-11-16 16:57:30 -05:00
weiminyu
dee559baee Define multiple test suites under Gradle for better test performance
Tests are divided into three test suites without intra-suite conflicts.
This allows us to unset forkEvery=1 (4x improvement on my desktop) and
increase execution parallelism (additional 2x improvement on my desktop).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221656972
2018-11-16 16:55:55 -05:00
guyben
6586460f3e Move AuthenticatedRegistrarAccessor to request/auth/
It is starting to be used in more places than just ur/server/registrar. Even now it's used in the RDAP, and we are going to start using it for the registrar-xhr endpoint meaning it will be used in EPP flows as well.

Also logically - this is part of the request authentication.

While moving - we also refactor it to make it easier to use in tests. Instead of mocking, we will be able to create instances with arbitrary roles.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221645055
2018-11-16 16:54:21 -05:00
mcilwain
b317aab22f Clean up some areas in preparation for externalized codebase
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221454055
2018-11-16 16:51:11 -05:00
jianglai
c0239b0a07 Move YamlUtils to be under google.registry.util package
This makes it simpler to package google.registry.util as a separate project in
Gradle that can be depended upon by the proxy package. Currently the proxy
package depends on both google.registry.util and google.registry.config.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221450085
2018-11-14 12:00:45 -05:00
guyben
e51cf3e9c7 Change OTE "fake" address state to a valid value
Currently, OTE creates an invalid address "state", which doesn't pass validation when trying to update it on the console.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221372006
2018-11-14 11:59:04 -05:00
weiminyu
75add42a1b Restore original System Properties after tests
Many registry tools tests modify system properties but do not
restore them to original state. These tests must be isolated
from each other and cannot share the same test execution process.

This has a huge impact on test performance under Gradle, which
seems to have higher process startup overhead. Current Gradle
test config has to set 'forEvery' to 1, i.e., every test class
must be run in a freshly started process.

This change significantly reduces the number of tests that need
isolation, making it easier to optimize test config for the
remaining tests.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221350284
2018-11-14 11:57:23 -05:00
jianglai
9fa2a84c35 Replace iteration over depsets with an explicit .to_list() call
The old pattern did an implicit iteration over a depset which will be forbidden in the future since it is potentially expensive. The new to_list() call is still expensive but it will be more visible.

LSC: []
Tested:
    TAP --sample for global presubmit queue
    []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221266626
2018-11-14 11:55:43 -05:00
guyben
4a31232423 Fix Kokoro build broken by []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221097556
2018-11-12 14:51:40 -05:00
shicong
66d98c8d66 Add SetNumInstancesCommand in Nomulus tool to adjust the number of instances
for a given service and version at runtime.

Note that this CL only supports the adjustment for a given service and version. I will add another functionality to let this command be able to detect all non-live versions automatically and apply the adjustment.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=221092001
2018-11-12 14:51:40 -05:00
guyben
557984bb75 Add support G-Suite group whose members have ADMIN access to registrar console
After this CL, "support" accounts (accounts that are part of the "support" G-Suite group) will the same access to the registrar console as GCP "admins". However, they don't won't have access to the GCP project itself.

We could give them their own Role in the future (say SUPPORT) and give them different access than "admins", but right now we don't need it and YAGNI or something :)

NOTE: we identify users by their email (they need to be logged in to a google account). I don't know if that's best practice, since I guess different google accounts might have the same email address. However, G-Suite groups' membership is by email so there's not much we can do about it if we want to use G-Suite groups.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220804273
2018-11-12 14:51:40 -05:00
weiminyu
783c010ab4 Add a IDE helper plugin for annotation processors
Without the net.ltgt.apt plugin, Dagger does not retain generated source code,
and AutoValue mixes generated source code with classes. The new plugin fixes this
problem, making it easier to work with IDEs.

Based on limit experience, Intellij Idea can properly import this project. We
may consider adding IDE-specific plugins in the future.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220802052
2018-11-12 14:51:40 -05:00
mmuller
9312cb3936 Don't use gradlew from travis
Since we're explicitly downloading a version of travis anyway, just run that
and don't bother with creating gradlew which does nothing but download the
same version over again.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220618139
2018-11-12 14:51:40 -05:00
guyben
61a5cf307e Add "Admin" tab to the registrar console
This tab will set the "allowedTlds", but might have other functionality in the
future.

It is based on (branches from) the security-settings tab, because I'm copying the functionality of the "whitelisted IPs" to the "allowed TLDs": they are both lists of "arbitrary" strings that you can remove from and add to.

There are a lot of moving parts in this CL, because of how all the different elements need to interact, and how intertwined they are (for example, we need to disable the admin-settings view for non admins both in the soy and in the JS code)

It's really time to refactor the console given all we've learned... :/

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220373443
2018-11-12 14:51:40 -05:00
jianglai
9b10c116f3 Do not create a logger during initialization in CidrAddressBlock
This is patched from [] We should have done this when we migrated to
Flogger.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219860046
2018-11-12 14:51:40 -05:00
guyben
9ce07db38a Make OutputEncapsulator a CommandRunner
This is in preparation for having other "command changing things" like
redirecting to file and maybe variable substitutions in the arguments.

"On the way" added a
RUNNING "some_command" "--some_flag" "some_value"
to the output encapsulator so that if we run multiple commands, we know what
command was called where.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219837452
2018-11-12 14:51:40 -05:00
Michael Muller
9b4d175471 Testing a direct "gradle build" (no gradlew) 2018-11-07 11:15:52 -05:00
mcilwain
f59005ad35 Fix sender email address for invoicing alerts
It was failing to send alert emails because the email address it was
constructing did not have permission through GAE to send emails. This switches
it over to using the send from email address already in use elsewhere in the app
that does successfully send emails.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219812019
2018-11-02 14:50:52 -04:00
weiminyu
6a870e5820 Check for unused dependencies
Enabled unused-dependency check using nebula-lint.

Dependencies that are not used by compile or testing are
labeled with 'maybe_runtime". We leave these dependencies
in the script for easy reference. Before launching Gradle-based
release process we must determine which of these should
be removed and which should be relabeled as runtime.

Label assignment:
- All dependencies recommended for removal from 'compile'
  are changed to maybe_runtime
- All dependencies recommended for move from 'compile' to
  testCompile are split into two lines, one with testCompile,
  the other maybe_runtime

Incidentally, Gradle 4.10.2 needs a groovy upgrade before
it can work with Oracle JDK 11.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219803797
2018-11-02 14:49:18 -04:00
jianglai
b5856a1467 Fix FOSS build
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219794867
2018-11-02 14:45:59 -04:00
guyben
3f6585fccc Reduce duplicate code in the servlets
Currently, all 4 servlets (backend, frontend, pubapi, tools) have duplicates of
the same exact code.

That's an anti-pattern!

Created a ServletBase they can all extend which has the duplicate code.

As a bonus, the tools servlet now runs the metric reporter, meaning tool
related metrics will now be reported!

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219792176
2018-11-02 14:44:22 -04:00
mmuller
545b68ad9a Add a .travis.yml file
Add the Travis-CI configuration file, which will allow our Gradle build to be
tested from Github with travis.

TESTED:
  1) Repeatedly tweaked this on own local fork of google/nomulus until I was able to successfully build.
  2) Verified that this shows up in the tarball after "blaze build :opensource-staging"

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219678030
2018-11-02 14:42:45 -04:00
guyben
08290e6b87 Fix the tool that was broken in []
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219629901
2018-11-02 14:39:36 -04:00
Michael Muller
79bbd5e29d Upped the build time to 45 minutes. 2018-11-01 08:48:07 -04:00
weiminyu
00d0284d5b Add nebula-lint plugin to Gradle scripts
Updated the project config so that the plugin can work.

Reverted to the deprecated compile/testCompile labels
so that the plugin can work. This plugin provides valuable
checks that we do not want to give up.

Added undeclared-dependency check. Still need to add
unused-dependency and one-version check.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219406600
2018-10-31 19:15:03 -04:00
mountford
09202562c7 RDAP: Add link to static TOS page
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219332975
2018-10-31 19:13:23 -04:00
guyben
57f06258d3 Add metrics measuring all request processing times
The cardinality of this new metric is:
buckets - 16
path + method - around 100 (the number of Actions we have)
authLevel - at most 3
success - 2

Total: 16*100*3*2 = 9,600

This is still low, especially for the value it could give in understanding our system (graphs of all endpoints, how often are they called, how long they take, how often do they fail)

Instead of "success true/false", we might want to give the actual status code. This can be a bit annoying because HttpServletResponse doesn't have a getStatus.

But it's possible, and worth considering.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219312400
2018-10-31 19:11:39 -04:00
guyben
a45d3d3bc7 Add a log statement at the very end of our code
Having a log at the very begining of "our" code helped us find issues with
App-Engine's dispatcher, where we could clearly see in the logs that "our" code
started many seconds after the request came in.

We now suspect there is something similar going on after the end of "our" code
- where the reply is sent back many seconds after our code finished running.

To make sure - we add a log statement at the very last line of "our" code, so
we know exactly when it ended.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219309962
2018-10-31 19:09:51 -04:00
Michael Muller
c1d586a211 Make travis wait up to 30 minutes on build without output.
Also add some comments to .travis.yml.
2018-10-31 17:13:16 -04:00
Michael Muller
beae72f6ba Set TERM=dumb in environment to limit build output
The status information gradle prints out doesn't look very good in Travis.  In
fact, it causes the logfile to exceed the maximum allowed by Travis.
2018-10-31 16:44:26 -04:00
Michael Muller
a3a474964c Do "gradlew build" insetead of just gradlew. 2018-10-31 16:00:18 -04:00
Michael Muller
1b543d55a5 Fixed url and unzip in gradle download. 2018-10-31 15:57:03 -04:00
Michael Muller
f5408b36e3 Attempt to install gradle on travis.
Try installing gradle instead of calling gradlew, which it turns out requires
its jarfiles to be presesnt in order to work.
2018-10-31 15:39:28 -04:00