We want to know how long it's actually taking to process asynchronous
contact/host deletions and DNS refreshes on host renames. This adds
instrumentation. Five metrics are recorded as follows:
* An incrementable metric for each async task processed (split out by
type of task and result).
* Two event metrics for processing time between when a task is enqueued
and when it is processed -- tracked separately for contact/host
deletion and DNS refresh on host rename.
* Two event metrics for batch size every time the two mapreduces are
run (this is usually 0). Tracked separately for contact/host deletion
and DNS refresh on host rename.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=157001310
We should be able to remove the dependency on the App Engine [] library from the frontend service, since no []s actually run there. In order to do this, we need to remove the various []-reliant classes from the frontend service build. This CL begins the process by moving the two async "flows" to a different package which is not included in the frontend service.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=142159568
A previous CL inadvertently caused the system to always set the transfer status to SERVER_CANCELLED when deleting a resource, even if there was no transfer. This led to RDE problems.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=140890919
Currently we pass in null. However, from the spec:
<domain:acDate> element that contains the date and time of a
required or completed response. For a PENDING request, the value
identifies the date and time by which a response is required
before an automated response action will be taken by the server.
For all other status types, the value identifies the date and time
when the request was completed."
- https://tools.ietf.org/html/rfc5731#page-16, section 3.1.3
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=139363370
This also changes the default number of mapper shards in tests to 2, which is
the number of EppResourceIndex buckets in unit tests. Running more shards than
there are buckets causes unnecessary test load.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=135520601
This applies lessons learned from the async batch DNS refresh action, in
particular making testing more robust.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134833523
This will replace the existing DnsRefreshForHostRenameAction.
This is stage one of a three stage migration process. It adds the new queue and
[] but doesn't call them yet. Stage two will cut over to using the new
functionality, and stage three will remove the old functionality.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134793963
It is replaced by loadByForeignKey(), which does the same thing that
loadByUniqueId() did for contacts, hosts, and domains, and also
loadDomainApplication(), which loads domain application by ROID. This eliminates
the ugly mode-switching of attemping to load by other foreign key or ROID.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133980156
Also consolidates the DNS refresh functionality in AsyncFlowUtils that was
being used by HostUpdateFlow into AsyncFlowEnqueuer.
TESTED=I threw together some batch scripts to create dozens of contacts on
alpha and then request their deletion, and the [] ran fine and
successfully deleted them in batches.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133714691
This allows handling of N asynchronous deletion requests simultaneously instead
of just 1. An accumulation pull queue is used for deletion requests, and the
async deletion [] is now fired off whenever that pull queue isn't empty,
and processes many tasks at once. This doesn't particularly take more time,
because the bulk of the cost of the async delete operation is simply iterating
over all DomainBases (which has to happen regardless of how many contacts and
hosts are being deleted).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133169336
This change replaces all Ref objects in the code with Key objects. These are
stored in datastore as the same object (raw datastore keys), so this is not
a model change.
Our best practices doc says to use Keys not Refs because:
* The .get() method obscures what's actually going on
- Much harder to visually audit the code for datastore loads
- Hard to distinguish Ref<T> get()'s from Optional get()'s and Supplier get()'s
* Implicit ofy().load() offers much less control
- Antipattern for ultimate goal of making Ofy injectable
- Can't control cache use or batch loading without making ofy() explicit anyway
* Serialization behavior is surprising and could be quite dangerous/incorrect
- Can lead to serialization errors. If it actually worked "as intended",
it would lead to a Ref<> on a serialized object being replaced upon
deserialization with a stale copy of the old value, which could potentially
break all kinds of transactional expectations
* Having both Ref<T> and Key<T> introduces extra boilerplate everywhere
- E.g. helper methods all need to have Ref and Key overloads, or you need to
call .key() to get the Key<T> for every Ref<T> you want to pass in
- Creating a Ref<T> is more cumbersome, since it doesn't have all the create()
overloads that Key<T> has, only create(Key<T>) and create(Entity) - no way to
create directly from kind+ID/name, raw Key, websafe key string, etc.
(Note that Refs are treated specially by Objectify's @Load method and Keys are not;
we don't use that feature, but it is the one advantage Refs have over Keys.)
The direct impetus for this change is that I am trying to audit our use of memcache,
and the implicit .get() calls to datastore were making that very hard.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131965491
This fixes a significant memory consumption issue when running tests.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129482832
ReferenceUnion is a hack to work around the mismatch between how
we store references (by roid) and how they are represented in EPP
(by foreign key). If it ever needed to exist (not entirely clear...)
it should have remained tightly scoped within the domain commands
and resources. Instead it has leaked everywhere in the project,
causing lots of boilerplate. This CL hides all of that behind
standard Refs, and should be followed by work to remove ReferenceUnion
completely.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=122424416
The dark lord Gosling designed the Java package naming system so that
ownership flows from the DNS system. Since we own the domain name
registry.google, it seems only appropriate that we should use
google.registry as our package name.
This change renames directories in preparation for the great package
rename. The repository is now in a broken state because the code
itself hasn't been updated. However this should ensure that git
correctly preserves history for each file.
