It broke because I forgot to add the new spec11 packages to gtld.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=206021827
This adds the scaffolding for a basic Spec11 pipeline- it gathers all domains from all time for a given project and counts how many there are. I've factored out a few common utilities for beam pipelines to avoid excessive duplication.
Future CLs will:
- Actually process domains via the SafeBrowsing API
- Generate a real spec11 report
- Template queries based on the input YearMonth
- Abstract more commonalities across beam pipelines to reduce boilerplate when adding new pipelines.
TESTED: FOSS test passed, and ran successfully on alpha
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205997741
This ensures that only one will run at a time, which should help fix the
clogged up mapreduces we've seen on sandbox.
In order to do this, the UnlockerOutput is introduced. This unlocks the
given Lock after all reducer shards have finished.
Also increases the lease duration of the DNS refresh action from 20 to
240 minutes. 20 minutes isn't long enough; when there's a lot of domains
and decent system load the mapreduce could take longer than that in the
ordinary case.
TESTED=Deployed to alpha and verified that more than one copy of the
mapreduce wouldn't run simultaneously, and also that the lock is
released when the mapreduce is finished.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205887554
The "file encoding" saves the file + metadata (filename and modification) in a "blob" format that PGP knows how to read.
Merges the file-encoder creation between RyDE and Ghostryde.
The new file - RydeFileEncoding.java - is a merge of the removed functions in
Ghostryde.java and the RydePgpFileOutputStream.java.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205295756
I'm finally fed up enough with all the nameserver changes we've had to make on our
self-allocated domains to improve the command. Now you can simply run:
$ nomulus ... update_domain ... -n ns[1-4].foo.bar
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205282317
Domains that are reserved with type NAME_COLLISION can be registered defensively
during sunrise only, but DNS can never resolve for them. Correspondingly, we
need to apply the SERVER_HOLD status for such registrations. We also send the
registrar a poll message informing them of this act.
This brings us up to feature parity with end-date sunrise (implemented in
DomainAllocateFlow), which already has all of this handling.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205277728
Merges the encryptor creation between RyDE and Ghostryde.
The new file - RydeEncryption.java - is a merge of the removed functions in
Ghostryde.java and the RydePgpEncryptionOutputStream.java.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205246053
Merges the compressor creation between RyDE and Ghostryde. Note that GhostRyde
will now compress with ZIP rather than the previous ZLIB. This is backwards
compatible because the decompression algorithm works with either, so files
created by the old version (with ZLIB) can still be opened by the new version,
and vice-versa.
The new file - RydeCompression.java - is a merge of the removed functions in Ghostryde.java and the RydePgpCompressionOutputStream.java.
This is one of a series of CLs - each merging a single "part" of the encoding.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205102150
Ghostryde.java has a lot of duplicate code with RydeEncoder and the future
RydeDecoder - the encryption/decryption, compression/decompression, file
encoding/decoding. The "de-XXX" part of each of these pairs needs to read a PGP
object from a stream using PGPObjectFactory.
Since we want to move the duplicate code into their own files, we will need to
move the "read PGP objects from stream" functions to a common utility class.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=205092800
We used to set the installer as executable on X20 and have kokoro copy it to
the temp folder and run it from there. Now that executables on x20 must be
built verifiable, we cannot set the +x bit any more.
Instead, run the script as an argument to the bash command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204536872
This prepares for the spec11 beam pipeline to live parallel to the invoicing
beam pipeline, for better organization.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204980582
Second step of RDE encoding refactoring.
Creates a single OutputStream encode RyDE files.
This replaces the 5 OutputStreams that were needed before.
Also removes all the factories that were injected. It's an encoding, there's no point in injecting it.
Finally, removed the buffer-size configuration and replaced with a static final
const value in each individual OutputStream.
This doesn't yet include a decoder (InputStream). And there's still a lot of overlap between the Ryde and the Ghostryde code. Both of those are left for the next CLs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204898369
This officially adds a 15% discount to sunrise creates and makes anchor tenant
creates free for the first 2 years.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204805141
Also goes back up to 100 max instances.
Hopefully this'll work better this time.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204783809
Lists used as accumulators were being updated individually for each domain
without starting over from a fresh list each time, so the number of changes
would grow for each additional domain and potentially be wrong if the previous
domains were set up differently.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204526006
This means that, when writing new tests that are failing, you get much more
useful logs that show the actual XML in a more comprehensible format that is
suitable for pasting back into the golden file in the test (if the change was
intended).
This requires outputting the standalone parameter in the XML transformer, and
some minor changes to some tests as a result that were relying on it being
stripped out.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204513690
This also introduces a production canary environment, similar to sandbox canary. The docker tags are changed to "live" and "sandbox" respectively, to reflect the fact that different images may be used for prod and sandbox.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204343530
This is a part of a large-scale change: [] . All .bzl files are being formatted with buildifier.
To format a file manually run `buildifier path/to/file.bzl`. Integration with `g4 fix` will be available later, but you can try using `g4 fix --format=bzl`.
Tested:
tap_presubmit
Some tests failed; test failures are believed to be unrelated to this CL
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204137752
The code stopped working after a recent soy-to-js compiler change.
Impacts include:
- The /registrar#domain page only shows part of the contents.
- RegistrarConsoleScreenshotTest times out.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=204135349
This is a part of a large-scale change: [] . All .bzl files are being formatted with buildifier.
To format a file manually run `buildifier path/to/file.bzl`. Integration with `g4 fix` will be available later, but you can try using `g4 fix --format=bzl`.
Tested:
tap_presubmit
Some tests failed; test failures are believed to be unrelated to this CL
BEGIN_PUBLIC
Format .bzl files with buildifier
END_PUBLIC
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=203461813
The action isn't routable (i.e. it doesn't work) unless this is specified.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=203020503
We never launched this, don't planning on launching it now anyway, and it's rotted over the past two years anyway.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202993577
Async tasks will now re-enqueue themselves after completion if there are
additional pending future actions. This allows all parts of domain delete flows
to be successfully re-saved as the parts happen, without going past the maximum
allowed 30 day task ETA limit. The first task runs at 30 days out when the
redemption grace period ends, and that task then enqueues another task to run 5
more days in the future, when the deletion is final and the pending delete
status gets removed.
No data migration plan is necessary because future resaves defaults to empty,
and indeed will always be empty on transfers. So previously enqueued tasks will
still be valid.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202949677
This removes the following unnecessary imports:
//third_party/java_src/gtld/java/google/registry/bigquery
//third_party/java_src/gtld/java/google/registry/config
//third_party/java_src/gtld/java/google/registry/cron
//third_party/java_src/gtld/java/google/registry/dns
//third_party/java_src/gtld/java/google/registry/gcs
//third_party/java_src/gtld/java/google/registry/mapreduce
//third_party/java_src/gtld/java/google/registry/model
//third_party/java_src/gtld/java/google/registry/module/backend
//third_party/java_src/gtld/java/google/registry/module/frontend
//third_party/java_src/gtld/java/google/registry/module/pubapi
//third_party/java_src/gtld/java/google/registry/module/tools
//third_party/java_src/gtld/java/google/registry/request
//third_party/java_src/gtld/java/google/registry/security
//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers
//third_party/java_src/gtld/java/google/registry/util
//third_party/java_src/gtld/java/google/registry/xjc
//third_party/java_src/gtld/javatests/google/registry/model
//third_party/java_src/gtld/javatests/google/registry/testing
//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java_src/gtld/java/google/registry/bigquery,//third_party/java_src/gtld/java/google/registry/config,//third_party/java_src/gtld/java/google/registry/cron,//third_party/java_src/gtld/java/google/registry/dns,//third_party/java_src/gtld/java/google/registry/gcs,//third_party/java_src/gtld/java/google/registry/mapreduce,//third_party/java_src/gtld/java/google/registry/model,//third_party/java_src/gtld/java/google/registry/module/backend,//third_party/java_src/gtld/java/google/registry/module/frontend,//third_party/java_src/gtld/java/google/registry/module/pubapi,//third_party/java_src/gtld/java/google/registry/module/tools,//third_party/java_src/gtld/java/google/registry/request,//third_party/java_src/gtld/java/google/registry/security,//third_party/java_src/gtld/java/google/registry/ui/soy/registrar:soy_java_wrappers,//third_party/java_src/gtld/java/google/registry/util,//third_party/java_src/gtld/java/google/registry/xjc,//third_party/java_src/gtld/javatests/google/registry/model,//third_party/java_src/gtld/javatests/google/registry/testing,//third_party/java_src/gtld/javatests/google/registry/testing/mapreduce'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202652421
Also adds Guy to list of CONTRIBUTORS and removes Xooglers (who definitely won't be contributing more code).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202476218
This removes the following unnecessary imports:
//third_party/java/activation
//third_party/java/bouncycastle
//third_party/java/bouncycastle_bcpg
//third_party/java/dagger
//third_party/java/dnsjava
//third_party/java/jaxws_api
//third_party/java/jcommander
//third_party/java/joda_money
//third_party/java/joda_time
//third_party/java/json_simple
//third_party/java/junit
//third_party/java/mockito
//third_party/java/re2j
//third_party/java/servlet/servlet_api
//third_party/java/truth:truth8
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/activation,//third_party/java/bouncycastle,//third_party/java/bouncycastle_bcpg,//third_party/java/dagger,//third_party/java/dnsjava,//third_party/java/jaxws_api,//third_party/java/jcommander,//third_party/java/joda_money,//third_party/java/joda_time,//third_party/java/json_simple,//third_party/java/junit,//third_party/java/mockito,//third_party/java/re2j,//third_party/java/servlet/servlet_api,//third_party/java/truth:truth8'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202344774
This removes the following unnecessary imports:
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='[],[],[],[],[],[],[],[],[],[],[],[],[],[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202325520
This affects JSR305, JSR330, and Guava annotations.
The exact command run to generate this CL was:
build_cleaner '//third_party/java_src/gtld/...' -c '' --dep_restrictions='//third_party/java/jsr330_inject,//third_party/java/jsr305_annotations,[]'
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202322747
First step of RDE encoding refactoring.
Creates a single InputStream (OutputStream) to decode (encode) Ghostryde files.
This replaces the 3 InputStreams (OutputStreams) that were needed before.
Also removes a lot of classes, and removes the "injection" of the Ghostryde
class. It's an encoding, there's no point in injecting it.
Finally, removed the buffer-size configuration and replaced with a static final
const value. It's just a buffer size - it doesn't actually affect much. There
are much more "important" fields that weren't configured (such as the
compression algorithm and whether or not to do integrity checks)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202319102
It only needs the error page HTML files; everything else isn't used by endpoints
served by the backend service and only serves to increase build times
(especially compiling all that JS).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202229514
We'll continue to use injected clocks for the rest of our tests that use signed marks files, so that they don't all fail after the current validity period. The new test TmchTestDataExpirationTest will let us know when the files are expired, so we can update them.
All updated test data files come from https://newgtlds.icann.org/en/about/trademark-clearinghouse/registries-registrars
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202208196
RdeUpload works on alpha and sandbox by sending the data to a google internal
server. It is still running (and succeeding) after each RdeStaging, but the
lack of the cron job means it is currently lagging behind.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202181148
Two main changes:
- Replaced getClass().getSimpleName() in the logs with a constructor-given name. Right now what we have is a lot of identical classes with slightly different names so that the logs would be different. With this change - we can later get rid of a lot of these classes and replace them with simple wrappers.
- Removed the "expected" feature. Only Tar uses it - and it can override onClose to do that (that's what it's there for!)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202129563
There's no reason to have to create 2,500 simulated database entities for a single test (which makes it take a really long time). Better to just set the relevant limit to be lower for testing purposes.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202120722
The signature has expired on Mon Jun 25 23:59:59 UTC 2018.
Instead of fixing all the test files with a new signed mark, we inject the
clock used to validate the signature.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202116721
In RFC 5730, clTrid is specified as optional. We ran into an error earlier this
year in which a registrar was not passing a client transaction id and we didn't
handle it correctly. So, this CL adds some tests of common EPP operations verify
that they work correctly when the clTrid is not specified.
This also slightly improves some flow logic to make it more obvious at first
glance that clTrid is indeed optional.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202000845
We never fully used this stuff but definitely no longer use it following our
recent billing refactor. It's confusing to retain all of these entities and
commands given that none of them are actually used by anything.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201978094
We've migrated everything over to using the standard SQL view now. The legacy
version is just causing confusion and costing us resources.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201966352
We haven't used this in years and I doubt it's still even functional. It's using
the legacy BigQuery tables and the pre-Registry 2.0 schema, for starters.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201960046
This is one last hanging piece of work left over from last year's Java 8
migration. There's no functionality changes in this CL, just refactoring.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201947600
It doesn't entirely make semantic sense, since the actual state of the
SystemClock isn't being preserved, but it makes injection into serializable
classes (e.g. mapreduces) much simpler, so it's worth doing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201755949
ICANN's Consistent Labeling & Display policy requires that these fields be
present for all domains. We're currently tracking down the last of the data on
our registrars so that we can display these, but if we don't have the data we
should still at least be displaying the field names to make it more clear that
we do support the field, we just don't have the data for this domain yet.
The two fields in question are:
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone:
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=201730290
