Commit graph

51 commits

Author SHA1 Message Date
mcilwain
1c4e79f99e Don't allow non-active registrars to create domains or applications
Specifically, this prevents suspended registrars from creating domains or applications. Pending registrars already can't perform these actions because they get an error message when attempting to log in.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170481338
2017-10-04 16:16:45 -04:00
larryruili
578673141c Migrate to new Sheets v4 api
This moves us from the oudated google/data XML api to the OnePlatform REST/JSON api, finally silencing the deprecation warnings we've been seeing.

The synchronization algorithm diffs the spreadsheet's current values with its internally sourced values, adding the row to a batch update request if there's a discrepancy. Additional internal data are added as an append operation to the end of the sheet, and any extraneous spreadsheet data is cleared from the spreadsheet.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=169273590
2017-09-20 10:27:17 -04:00
jianglai
c3f2002e6c Set up github CI kokoro job
TESTED=Set up the job in kokoro-qa:

[]

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166875896
2017-09-20 10:27:17 -04:00
guyben
978149e677 Create an injectable LockHandler
We create an injectable LockHandler that just calls the static
Lock.executeWithLocks function.

I'm not sure what's the correct place to put the LockHandler. I think
model/server is only appropriate for the actual datastore lock. This is a "per request" lock, so maybe request/lock?

-----------------------------

This is the initial step in adding the "lock implicitly released on request death" feature, but it's also useful on its own - easier to test Actions when we can use a fake lock.

To keep this CL simple, we keep using the old Lock as is in most places. We just choose a single example to convert to LockHandler to showcase it. Converting all other uses will be in a subsequent CL.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167357564
2017-09-12 15:51:50 -04:00
larryruili
63fdb506df Add explicit useLegacySql flag to UpdateSnapshotViewAction
It turns out the Bigquery JSON api selects its validator exclusively through
the useLegacySql flag (the #standardSQL directive isn't considered). To fix
this, we add back the explicit flag.

This also logs unexpected API errors, instead of allowing the job to quietly fail.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=166757569
2017-08-29 17:20:09 -04:00
guyben
aee4f7acc2 Remove queueing from Lock
It was buggy (didn't work) and was never actually used.

Why never actually used: for it to be used executeWithLock has to be called
with different requesters on the same lockId. That never happend in the code.

How it was buggy: Logically, the queue is deleted on release of the lock (meaning it was
meaningless the only time it mattered - when the lock isn't taken). In
addition, a different bug meant that having items in the queue prevented the
lock from being released forcing all other tasks to have to wait for lock
timeout even if the task that acquired the lock is long done.

Alternative: fix the queue. This would mean we don't want to delete the lock on release (since we want to keep the queue). Instead, we resave the same lock with expiration date being START_OF_TIME. In addition - we need to fix the .equals used to determine if the lock the same as the acquired lock - instead use some isSame function that ignores the queue.

Note: the queue is dangerous! An item (calling class / action) in the first place of a queue means no other calling class can get that lock. Everything is waiting for the first calling class to be re-run - but that might take a long time (depending on that action's rerun policy) and even might never happen (if for some reason that action decided it was no longer needed without acquiring the lock) - causing all other actions to stall forever!

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163705463
2017-08-01 17:06:20 -04:00
larryruili
d2cd576796 Add standardSQL views to Bigquery Datastore snapshots
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=163124895
2017-08-01 17:03:28 -04:00
mcilwain
d536cef20f Make Registrar load methods return Optionals instead of Nullables
This makes the code more understandable from callsites, and also forces
users of this function to deal with the situation where the registrar
with a given client ID might not be present (it was previously silently
NPEing from some of the callsites).

This also adds a test helper method loadRegistrar(clientId) that retains
the old functionality for terseness in tests. It also fixes some instances
of using the load method with the wrong cachedness -- some uses in high-
traffic situations (WHOIS) that should have caching, but also low-traffic
reporting that don't benefit from caching so might as well always be
current.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162990468
2017-08-01 16:58:59 -04:00
guyben
e224a67eda Change @Auth to an AutoValue, and created a set of predefined Auths
We want to be safer and more explicit about the authentication needed by the many actions that exist.

As such, we make the 'auth' parameter required in @Action (so it's always clear who can run a specific action) and we replace the @Auth with an enum so that only pre-approved configurations that are aptly named and documented can be used.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=162210306
2017-08-01 16:33:10 -04:00
jianglai
0013312f5c Export billing account map to registrar sheet
The billing account map will be serialized in the following format:

{currency1=id1, currency2=id2, ...}

In order for the output to be deterministic, the billing account map is stored as a sorted map.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161075814
2017-07-10 11:37:23 -04:00
mountford
7d2f53a6fe Remove xsrfScope and xsrfProtection authentication attributes
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=159121132
2017-06-21 09:46:40 -04:00
cgoldfeder
c9d7e75946 Cache Registrars in memory
This replaces the memcache caching, which we think is overall a bad idea.
We load all registrars at once instead of caching each as needed, so that
the loadAllCached() methods can be cached as well, and therefore will
always produce results consistent with loadByClientIdCached()'s view of the
registrar's values. All of our prod registrars together total 300k of data
right now, so this is hardly worth optimizing further, and in any case this
will likely reduce latency even further since most requests will be
served out of memory.

While I was in the Registrar file I standardized the error messages for incorrect
password and clientId length to be the same format, and cleaned up a few
random things I noticed in the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156151828
2017-05-17 12:26:11 -04:00
jianglai
5313ca58d6 Setup auth settings in PublishDetailReports @Action to support tool access
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=155459413
2017-05-17 11:43:09 -04:00
mcilwain
b03bd3b525 Remove UNRESERVED as a reservation type
This is a follow-up to Lai's refactoring of the get reservation types
code to return a set rather than a single type. Since we're always
returning a set now, the more natural way to represent a label that is
not reserved is to return an empty set rather than a set containing
UNRESERVED.

Also fixes some minor style issues I ran across regarding static
importing and test method naming that I ran across (no logic
implications).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=151132116
2017-03-27 13:31:33 -04:00
mcilwain
cdadb54acd Refer to Datastore everywhere correctly by its capitalized form
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=147479683
2017-02-17 12:12:12 -05:00
mcilwain
d6e6e8a49c Re-add some accidentally removed license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146694719
2017-02-06 16:45:23 -05:00
mmuller
b70f57b7c7 Update copyright year on all license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146111211
2017-02-02 16:27:22 -05:00
jianglai
4fed3a9ae6 Daggerize ExportSnapshotServlet and CheckSnapshotServlet
Eradicate the last remnants of un-injectable servlets!

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145598002
2017-01-30 15:03:53 -05:00
mcilwain
f647ea1190 Move registry policy settings and some others into YAML config
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145571850
2017-01-30 15:03:53 -05:00
mcilwain
eaec03e670 Move ConfigModule and LocalTestConfig into RegistryConfig
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
2017-01-09 12:01:09 -05:00
mcilwain
c5c74961bb Make all but one field on RegistryConfig static
The next step will be to get rid of RegistryConfig descendants and RegistryConfigLoader entirely.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143812815
2017-01-09 12:01:09 -05:00
mcilwain
25a8bbe890 Deprecate more fields in RegistryConfig
This primarily addresses issues with TMCH testing mode and email sending utils.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143710550
2017-01-09 11:59:04 -05:00
jart
734130aa73 Restructure Maven dependencies in build
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072

We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018

A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55

The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.

Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:

- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
2017-01-09 11:59:04 -05:00
mcilwain
28f6c770c8 Add MOE equivalence for sync on 2016-12-19
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142449539
2017-01-09 11:59:04 -05:00
Tom Johnson
b0ebeed5a5 Add @ReportedOn annotation for BigQuery exports
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142038227
2016-12-19 11:09:11 -05:00
mcilwain
58b7babc2b Fix Java 7 generic type inference issue in SyncGroupMembersAction
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141485405
2016-12-13 15:56:16 -05:00
mcilwain
2620097599 Retry attempted syncs to Google Groups that fail
I also moved to a non-concurrent modification syncing model. It was adding more
complexity than was justified just to have two requests going simultaneously
instead of one. The API doesn't reliably allow much more than that anyway.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141210192
2016-12-07 15:19:43 -05:00
mcilwain
2b7d580bb3 Run buildifier on codebase to format BUILD files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140362453
2016-11-28 18:15:21 -05:00
mcilwain
274b070b54 Make driveFolderId on Registry entity optional
This defaults to null, and leaving it to null now simply disables reserved terms
exporting, rather than throwing an error every time the action runs.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=138763161
2016-11-15 10:38:20 -05:00
mcilwain
cef07f6bc5 Use cursor to track updating of registrar sheet
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=138221931
2016-11-10 11:21:29 -05:00
ctingue
b7a98451e6 Add retry logic for SheetSynchronizer
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=138211996
2016-11-10 11:18:25 -05:00
jianglai
59d998954c Use correct <a> tag syntax in javadoc @see tag
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137946021
2016-11-02 15:19:34 -04:00
jart
2e81de9954 Make essential Bazel packages publicly visible
This allows separate Bazel projects to reference Nomulus as an external
repository. They can then copy the []
directory structure into their own project and customize the Action
and Module lists for the GAE modules in their own deployment.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136863886
2016-10-24 11:57:00 -04:00
ctingue
1f5a8d5542 Fix off-by-one issue in SheetSynchronizer
This doesn't change the end result of a successful run, though this is what a typical flow looks like prior to this fix:

Consider a sheet with 10 data rows (+ 1 header row = 11). A 10-row data set will call worksheet.setRowCount(10), which truncates the last row of the existing sheet. This row will eventually be added again in the last for loop, but if the synchronizer fails mid-sync, this last row will remain dropped. This fix will prevent this last row from being dropped.

This doesn't fix the broader issue of SheetSynchronizer not behaving transactionally -- that's a different can of worms.

See the linked bug for an instance where the synchronizer failed mid-run and dropped a data row as a result.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136398109
2016-10-17 18:06:43 -04:00
ctingue
d9dbf2ec1a Fix spelling in Javadoc for SheetSynchronizer
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136387503
2016-10-17 18:00:59 -04:00
ctingue
dd9137fbd1 Rename LrpToken to LrpTokenEntity
This is to better distinguish between an LRP "token" (the string passed along in EPP) and the datastore entity that contains the token and all metadata.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135943480
2016-10-14 16:58:07 -04:00
shikhman
f76bc70f91 Preserve test logs and test summary output for Kokoro CI runs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135494972
2016-10-14 16:57:43 -04:00
nickfelt
0d122acec2 Update some internal build configuration options
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135014718
2016-10-04 09:58:26 -04:00
mcilwain
ffcbdbf37b Set method to POST on actions called from cron fanout
The default is to support GET, which doesn't work with cron fanout which only
uses POST.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134284855
2016-09-26 13:50:03 -04:00
mcilwain
4813ed392b Rename "clientIdentifier" to "clientId" almost everywhere
It's best to be consistent and use the same thing everywhere.  "clientId" was
already used in more places and is shorter and no more ambiguous, so it's the
logical one to win out.

Note that this CL is almost solely a big Eclipse-assisted refactoring. There are
two places that I did not change clientIdentifier -- the actual entity field on
Registrar (though I did change all getters and setters), and the name of a
column on the exported registrar spreadsheet. Both would require data
migrations.

Also fixes a few minor nits discovered in touched files, including an incorrect
test in OfyFilterTest.java and some superfluous uses of String.format() when
calling checkArgument().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133956465
2016-09-22 14:30:05 -04:00
ctingue
c75ab37dab Export LrpToken to BigQuery
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132083818
2016-09-07 11:51:28 -04:00
Justine Tunney
37d30591ed Narrowly scope privileges for API service objects
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129099520
2016-08-02 19:18:09 -04:00
Justine Tunney
7f3f03ee97 MOE strip compatible_with
This is an internal-only feature that breaks the open source build.

CL created with:

    dr-replace '(compatible_with.*)' '\1  # MOE:strip_line'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=128852873
2016-08-02 19:14:28 -04:00
Chris Povirk
5332ac4e4a Set compatible_with=appengine on GAE targets
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=128475519
2016-08-02 19:09:11 -04:00
mountford
e72491e59b Replace to(Upper|Lower)Case with Ascii.to$1Case
The presubmits are warning that toUpperCase() and toLowerCase() are locale-specific, and advise using Ascii.toUpperCase() and Ascii.toLowerCase() as a local-invariant alternative.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=127583677
2016-07-19 10:43:38 -04:00
mcilwain
aa2f283f7c Convert entire project to strict lexicographical import sort ordering
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=127234970
2016-07-13 15:59:53 -04:00
nickfelt
038d3d5031 Fix latest_snapshot "pointer" query to use to fully-qualified table reference
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125105776
2016-06-27 16:09:10 -04:00
Corey Goldfeder
68b75461db Delete the useless MapreduceAction interface
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=122630519
2016-05-18 13:09:40 -04:00
mcilwain
b60ceda055 Migrate @Provides(type=SET_VALUES) to @ElementsIntoSet
See Rosie [] for context.

We've already switched over to using Dagger 2.4 in respositories.bzl,
so this change is fine for our open source drop.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=121863923
2016-05-13 23:19:48 -04:00
Michael Muller
c458c05801 Rename Java packages to use the .google TLD
The dark lord Gosling designed the Java package naming system so that
ownership flows from the DNS system. Since we own the domain name
registry.google, it seems only appropriate that we should use
google.registry as our package name.
2016-05-13 20:04:42 -04:00