Commit graph

97 commits

Author SHA1 Message Date
Wolfgang Meyers
60a3d2a53b Synchronize domains with DNS on import
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146813124
2017-02-08 17:42:43 -05:00
Wolfgang Meyers
a904f2c6ee Add mapper to import domains from RDE deposits
With some additional cleanup by Ben McIlwain.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146695665
2017-02-06 16:46:46 -05:00
cgoldfeder
1577ab2c26 Remove an unused exception
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146258125
2017-02-02 17:04:06 -05:00
mmuller
b70f57b7c7 Update copyright year on all license headers
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=146111211
2017-02-02 16:27:22 -05:00
cgoldfeder
0b1781b110 Make LINKED into a virtual status value
* Remove LINKED when loading an EppResource
* Enforce that you can't add it to a resource
* Ignore LINKED on xjc import of contacts and hosts

After running ResaveAllEppResourcesAction we will no
longer have persisted LINKED statuses in datastore.

In the process of writing this I discovered that RDAP
treats LINKED like any other status value and returns
the persisted value rather than the derived one. Since
this is an existing bug and is orthogonal to the changes
in this CL, I am addressing it in a separate CL.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145585227
2017-01-30 15:03:53 -05:00
mcilwain
eaec03e670 Move ConfigModule and LocalTestConfig into RegistryConfig
This is the final preparatory step necessary in order to load and load
configuration from YAML in a static context and then provide it either via
Dagger (using ConfigModule) or through RegistryConfig's existing static
functions.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143819983
2017-01-09 12:01:09 -05:00
mcilwain
c5c74961bb Make all but one field on RegistryConfig static
The next step will be to get rid of RegistryConfig descendants and RegistryConfigLoader entirely.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143812815
2017-01-09 12:01:09 -05:00
jart
734130aa73 Restructure Maven dependencies in build
We're now using java_import_external instead of maven_jar. This allows
us to specify the relationships between jars, thereby allowing us to
eliminate scores of vendor BUILD files that did nothing but re-export
@foo//jar targets, thus addressing the concerns of djhworld on Hacker
News: https://news.ycombinator.com/item?id=12738072

We now have redundant failover mirrors, which is a feature I added to
Bazel 0.4.2 in ed7ced0018

A new standard naming convention is now being used for all Maven repos.
Those names are calculated from the group_artifact name using the
following algorithm that eliminates redundancy:
https://gist.github.com/jart/41bfd977b913c2301627162f1c038e55

The JSR330 dep has been removed from java targets if they also depend
on Dagger, since Dagger always exports JSR330.

Annotation processor dependencies should now be leaner and meaner, by
more appropriately managing what needs to be on the classpath at
runtime. This should trim down the production jar by >1MB. As it stands
currently in the open source world:

- backend_jar_deploy.jar: 50MB
- frontend_jar_deploy.jar: 30MB
- tools_jar_deploy.jar: 45MB

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=143487929
2017-01-09 11:59:04 -05:00
mcilwain
28f6c770c8 Add MOE equivalence for sync on 2016-12-19
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142449539
2017-01-09 11:59:04 -05:00
mcilwain
a56a0677ba Refactor RDE import stuff into its own rde.imports package
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141566236
2016-12-13 15:57:46 -05:00
mountford
50aad72383 Add retry logic to RDE upload
We experience intermittent exceptions when connecting to Iron Mountain. Just retrying is probably the easiest solution to the problem.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141221179
2016-12-07 15:19:43 -05:00
Wolfgang Meyers
f2faf49d58 Add XjcToDomainResourceConverter for RDE import
With some code cleanup/refactoring/formatting by Ben McIlwain.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141199855
2016-12-07 15:19:29 -05:00
mountford
fbbcf41503 Temporarily change RDE to ignore bad transfer data
Because RDE reads from the commit logs to get its data, the bad transfer data resulting from a bug in the system is currently preventing proper RDE operation even for domains which have been fixed in Datastore. This CL adds a check to see if the transfer data is messed up (i.e. it doesn't have valid gaining and losing registrars), and if so, omits it.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=141074673
2016-12-06 11:52:46 -05:00
mountford
da711343f0 Improve RDE error logging
RDE logs somewhat inscrutable errors when XML validation of the output fails due to bad data in Datastore. This CL adds a little more information to help us track down this sort of problem in the future.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140872462
2016-12-06 11:52:46 -05:00
mcilwain
2b7d580bb3 Run buildifier on codebase to format BUILD files
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140362453
2016-11-28 18:15:21 -05:00
cgoldfeder
f5d0de61f2 Fix bad javadoc
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140389732
2016-11-28 17:43:10 -05:00
mmuller
ddacd33c5e Retry RDE report on SocketTimeoutException
It's likely that either 1) we should be doing this for more than a
SocketTimeoutException - we probably want to do this in the case of a number
of different transient failures 2) we don't want to do this at all because it
happens in a background task that will get re-run anyway.

In any case, this seems like the right fix and it addresses a problem we see
occassionally.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=139935190
2016-11-28 11:31:16 -05:00
jianglai
59d998954c Use correct <a> tag syntax in javadoc @see tag
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137946021
2016-11-02 15:19:34 -04:00
jart
2e81de9954 Make essential Bazel packages publicly visible
This allows separate Bazel projects to reference Nomulus as an external
repository. They can then copy the []
directory structure into their own project and customize the Action
and Module lists for the GAE modules in their own deployment.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136863886
2016-10-24 11:57:00 -04:00
Wolfgang Meyers
4cfe107475 Add mapreduce for RDE hosts import
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136766682
2016-10-24 11:56:53 -04:00
ctingue
67da55536f Clean up misc Javadoc and deprecation warnings
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136156922
2016-10-14 17:00:33 -04:00
Wolfgang Meyers
94af94ddff Refactor some RDE import logic to be generic
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136076610
2016-10-14 17:00:26 -04:00
shikhman
f76bc70f91 Preserve test logs and test summary output for Kokoro CI runs
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135494972
2016-10-14 16:57:43 -04:00
mcilwain
3b02d77ceb Rename 'registry_tool' to 'nomulus'
This changes everything with external visibility beyond the codebase
(i.e. the name of the compiled binary and the documentation that refers
to it). It does not change a lot of things internal to the codebase,
i.e. the "RegistryTool" class didn't change its name. We can rename that
in a subsequent CL if we want to.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135022087
2016-10-04 09:59:54 -04:00
mcilwain
b7a2c36be8 Fix all uses of DateTime.now() to use DateTimeZone.UTC
Almost all uses were in test classes, which I replaced with clock.nowUTC().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134993149
2016-10-03 16:43:53 -04:00
mcilwain
d853e59c7f Cut over to batched DNS refreshing on host renames
TESTED=I deployed it on alpha, renamed some hosts, and verified that
the [] ran as expected.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134991941
2016-10-03 16:42:24 -04:00
mcilwain
4813ed392b Rename "clientIdentifier" to "clientId" almost everywhere
It's best to be consistent and use the same thing everywhere.  "clientId" was
already used in more places and is shorter and no more ambiguous, so it's the
logical one to win out.

Note that this CL is almost solely a big Eclipse-assisted refactoring. There are
two places that I did not change clientIdentifier -- the actual entity field on
Registrar (though I did change all getters and setters), and the name of a
column on the exported registrar spreadsheet. Both would require data
migrations.

Also fixes a few minor nits discovered in touched files, including an incorrect
test in OfyFilterTest.java and some superfluous uses of String.format() when
calling checkArgument().

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133956465
2016-09-22 14:30:05 -04:00
Hans Ridder
2d46c7c27c Make RDE SSH key identity injectable
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133883090
2016-09-22 14:29:55 -04:00
Wolfgang Meyers
28eeda189d Add map reduce job for contact import
Some additional changes were made by Ben McIlwain.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133875824
2016-09-22 14:29:20 -04:00
cgoldfeder
5098b03af4 DeReference the codebase
This change replaces all Ref objects in the code with Key objects. These are
stored in datastore as the same object (raw datastore keys), so this is not
a model change.

Our best practices doc says to use Keys not Refs because:
 * The .get() method obscures what's actually going on
   - Much harder to visually audit the code for datastore loads
   - Hard to distinguish Ref<T> get()'s from Optional get()'s and Supplier get()'s
 * Implicit ofy().load() offers much less control
   - Antipattern for ultimate goal of making Ofy injectable
   - Can't control cache use or batch loading without making ofy() explicit anyway
 * Serialization behavior is surprising and could be quite dangerous/incorrect
   - Can lead to serialization errors. If it actually worked "as intended",
     it would lead to a Ref<> on a serialized object being replaced upon
     deserialization with a stale copy of the old value, which could potentially
     break all kinds of transactional expectations
 * Having both Ref<T> and Key<T> introduces extra boilerplate everywhere
   - E.g. helper methods all need to have Ref and Key overloads, or you need to
     call .key() to get the Key<T> for every Ref<T> you want to pass in
   - Creating a Ref<T> is more cumbersome, since it doesn't have all the create()
     overloads that Key<T> has, only create(Key<T>) and create(Entity) - no way to
     create directly from kind+ID/name, raw Key, websafe key string, etc.

(Note that Refs are treated specially by Objectify's @Load method and Keys are not;
we don't use that feature, but it is the one advantage Refs have over Keys.)

The direct impetus for this change is that I am trying to audit our use of memcache,
and the implicit .get() calls to datastore were making that very hard.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131965491
2016-09-02 13:50:20 -04:00
Wolfgang Meyers
d3d33ccfdc Add escrow file import validation logic
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129116064
2016-08-02 19:22:30 -04:00
Justine Tunney
e3bb5dc9b0 Fix up license headers and Python linting
This will help [] be submitted without breaking the linter.
License headers are now added automatically where they were previously
added by hand. We're also now adding the license header to Soy and SQL
files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129017424
2016-08-02 19:16:42 -04:00
Justine Tunney
7f3f03ee97 MOE strip compatible_with
This is an internal-only feature that breaks the open source build.

CL created with:

    dr-replace '(compatible_with.*)' '\1  # MOE:strip_line'

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=128852873
2016-08-02 19:14:28 -04:00
Chris Povirk
5332ac4e4a Set compatible_with=appengine on GAE targets
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=128475519
2016-08-02 19:09:11 -04:00
Wolfgang Meyers
ac6b147bf5 Add ContactResource import utility logic
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=127765049
2016-07-19 10:48:47 -04:00
mountford
e72491e59b Replace to(Upper|Lower)Case with Ascii.to$1Case
The presubmits are warning that toUpperCase() and toLowerCase() are locale-specific, and advise using Ascii.toUpperCase() and Ascii.toLowerCase() as a local-invariant alternative.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=127583677
2016-07-19 10:43:38 -04:00
mcilwain
aa2f283f7c Convert entire project to strict lexicographical import sort ordering
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=127234970
2016-07-13 15:59:53 -04:00
Wolfgang Meyers
bdfa97b0ae Add XjcToContactResourceConverter
XjcToContactResourceConverter is, as it name suggests, an inverse of
ContactResourceToXjcConverter. This utility class is designed to
support the TLD data import feature.

EXTERNAL_REVIEW_URL=https://github.com/google/domain-registry/pull/19
GIT_AUTHOR=Wolfgang Meyers <wolfgang@donuts.co>
(With some minor changes by Ben McIlwain.)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125985714
2016-07-01 11:38:05 -04:00
ctingue
14522eac0c Cut RegistryCursor over to global cursors
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=125797729
2016-06-27 16:44:35 -04:00
cgoldfeder
ec39f15a23 Fix generics in EppXmlTransformer.unmarshal to not be only on the return type.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=124914271
2016-06-27 15:55:03 -04:00
Wolfgang Meyers
81dc55ceab Add RdeParser
RdeParser can parse an escrow deposit file from a stream without
loading the entire file into memory, but offers the convenience
of parsing RDE elements into their jaxb object representation.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=124142087
2016-06-06 13:31:49 -04:00
Corey Goldfeder
d9875ea302 Remove unused and misused methods from EppResourceUtils and DomainBase.
The methods in DomainBase were the only callers for the methods in
EppResourceUtils, so I first inlined them. Then I realized that there
were no callers for loadReferencedContacts() anywhere. For loadNameservers(),
all but one invocation actually wanted to load the foreign keys, and was
repeating that work, so I replaced it with loadNameserverFullyQualifiedHostNames().
The last invocation, in the Rdap code, was incorrectly assuming this was an async
load when in fact it blocks, so I replaced it with a real async load.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=122433897
2016-05-16 18:39:23 -04:00
cgoldfeder
9a2afc7a9b Remove nearly all uses of ReferenceUnion
ReferenceUnion is a hack to work around the mismatch between how
we store references (by roid) and how they are represented in EPP
(by foreign key). If it ever needed to exist (not entirely clear...)
it should have remained tightly scoped within the domain commands
and resources. Instead it has leaked everywhere in the project,
causing lots of boilerplate. This CL hides all of that behind
standard Refs, and should be followed by work to remove ReferenceUnion
completely.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=122424416
2016-05-16 16:36:25 -04:00
Ben McIlwain
8f986d6f55 Get rid of deprecated rde-staging queue
It's no longer used by the current RDE system.  Also, remove an unused
Dagger @PRovides method for the rde-upload queue.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=122164826
2016-05-13 23:25:54 -04:00
jart
6082146d55 Prefer re2 over java.util.regex
Java's stock regex implementation doesn't guarantee linear time
complexity which makes it a security liability.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=121159875
2016-05-13 23:16:07 -04:00
Michael Muller
c458c05801 Rename Java packages to use the .google TLD
The dark lord Gosling designed the Java package naming system so that
ownership flows from the DNS system. Since we own the domain name
registry.google, it seems only appropriate that we should use
google.registry as our package name.
2016-05-13 20:04:42 -04:00
Justine Tunney
5012893c1d mv com/google/domain/registry google/registry
This change renames directories in preparation for the great package
rename. The repository is now in a broken state because the code
itself hasn't been updated. However this should ensure that git
correctly preserves history for each file.
2016-05-13 18:55:08 -04:00