zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-06-30 00:03:37 +02:00
Code Issues Projects Releases Packages Wiki Activity

Remove unused dns-cron queue

Browse source 
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132884314
This commit is contained in:
shikhman 2016-09-12 09:49:42 -07:00 • committed by Ben McIlwain
parent b9b2829f7c
commit cee08d48f2
2 changed files with 72 additions and 83 deletions
Download patch file Download diff file Expand all files Collapse all files

55
g3doc/app-engine-architecture.md
View file

@ -109,40 +109,41 @@ explicitly marked as otherwise.
streamed to BigQuery in the `Metrics` class. Tasks are enqueued during EPP streamed to BigQuery in the `Metrics` class. Tasks are enqueued during EPP
flows in `EppController`. This means that there is a lag of a few seconds to flows in `EppController`. This means that there is a lag of a few seconds to
a few minutes between when metrics are generated and when they are queryable a few minutes between when metrics are generated and when they are queryable
in BigQuery, but this is preferable to slowing all EPP flows down and blocking in BigQuery, but this is preferable to slowing all EPP flows down and
them on BigQuery streaming. blocking them on BigQuery streaming.
* `brda` -- Queue for tasks to upload weekly Bulk Registration Data Access * `brda` -- Queue for tasks to upload weekly Bulk Registration Data Access
(BRDA) files to a location where they are available to ICANN. The (BRDA) files to a location where they are available to ICANN. The
`RdeStagingReducer` (part of the RDE MapReduce) creates these tasks at the end `RdeStagingReducer` (part of the RDE MapReduce) creates these tasks at the
of generating an RDE dump. end of generating an RDE dump.
* `delete-commits` -- Cron queue for tasks to regularly delete commit logs that * `delete-commits` -- Cron queue for tasks to regularly delete commit logs
are more than thirty days stale. These tasks execute the that are more than thirty days stale. These tasks execute the
`DeleteOldCommitLogsAction`. `DeleteOldCommitLogsAction`.
* `dns-cron` (cron queue) and `dns-pull` (pull queue) -- A push/pull pair of * `dns-pull` -- A pull queue to enqueue DNS modifications. Cron regularly runs
queues. Cron regularly enqueues tasks in dns-cron each minute, which are then `ReadDnsQueueAction`, which drains the queue, batches modifications by TLD,
executed by `ReadDnsQueueAction`, which leases a batch of tasks from the pull and writes the batches to `dns-publish` to be published to the configured
queue, groups them by TLD, and writes them as a single task to `dns-publish` `DnsWriter` for the TLD.
to be published to the configured DNS writer for the TLD. * `dns-publish` -- Queue for batches of DNS updates to be pushed to DNS
* `dns-publish` -- Queue for batches of DNS updates to be pushed to DNS writers. writers.
* `export-bigquery-poll` -- Queue for tasks to query the success/failure of a * `export-bigquery-poll` -- Queue for tasks to query the success/failure of a
given BigQuery export job. Tasks are enqueued by `BigqueryPollJobAction`. given BigQuery export job. Tasks are enqueued by `BigqueryPollJobAction`.
* `export-commits` -- Queue for tasks to export commit log checkpoints. Tasks * `export-commits` -- Queue for tasks to export commit log checkpoints. Tasks
are enqueued by `CommitLogCheckpointAction` (which is run every minute by are enqueued by `CommitLogCheckpointAction` (which is run every minute by
cron) and executed by `ExportCommitLogDiffAction`. cron) and executed by `ExportCommitLogDiffAction`.
* `export-reserved-terms` -- Cron queue for tasks to export the list of reserved * `export-reserved-terms` -- Cron queue for tasks to export the list of
terms for each TLD. The tasks are executed by `ExportReservedTermsAction`. reserved terms for each TLD. The tasks are executed by
`ExportReservedTermsAction`.
* `export-snapshot` -- Cron and push queue for tasks to load a Datastore * `export-snapshot` -- Cron and push queue for tasks to load a Datastore
snapshot that was stored in Google Cloud Storage and export it to BigQuery. snapshot that was stored in Google Cloud Storage and export it to BigQuery.
Tasks are enqueued by both cron and `CheckSnapshotServlet` and are executed by Tasks are enqueued by both cron and `CheckSnapshotServlet` and are executed
both `ExportSnapshotServlet` and `LoadSnapshotAction`. by both `ExportSnapshotServlet` and `LoadSnapshotAction`.
* `export-snapshot-poll` -- Queue for tasks to check that a Datastore snapshot * `export-snapshot-poll` -- Queue for tasks to check that a Datastore snapshot
has been successfully uploaded to Google Cloud Storage (this is an has been successfully uploaded to Google Cloud Storage (this is an
asynchronous background operation that can take an indeterminate amount of asynchronous background operation that can take an indeterminate amount of
time). Once the snapshot is successfully uploaded, it is imported into time). Once the snapshot is successfully uploaded, it is imported into
BigQuery. Tasks are enqueued by `ExportSnapshotServlet` and executed by BigQuery. Tasks are enqueued by `ExportSnapshotServlet` and executed by
`CheckSnapshotServlet`. `CheckSnapshotServlet`.
* `export-snapshot-update-view` -- Queue for tasks to update the BigQuery views * `export-snapshot-update-view` -- Queue for tasks to update the BigQuery
to point to the most recently uploaded snapshot. Tasks are enqueued by views to point to the most recently uploaded snapshot. Tasks are enqueued by
`LoadSnapshotAction` and executed by `UpdateSnapshotViewAction`. `LoadSnapshotAction` and executed by `UpdateSnapshotViewAction`.
* `flows-async` -- Queue for asynchronous tasks that are enqueued during EPP * `flows-async` -- Queue for asynchronous tasks that are enqueued during EPP
command flows. Currently all of these tasks correspond to invocations of any command flows. Currently all of these tasks correspond to invocations of any
@ -151,15 +152,15 @@ explicitly marked as otherwise.
* `group-members-sync` -- Cron queue for tasks to sync registrar contacts (not * `group-members-sync` -- Cron queue for tasks to sync registrar contacts (not
domain contacts!) to Google Groups. Tasks are executed by domain contacts!) to Google Groups. Tasks are executed by
`SyncGroupMembersAction`. `SyncGroupMembersAction`.
* `load[0-9]` -- Queues used to load-test the system by `LoadTestAction`. These * `load[0-9]` -- Queues used to load-test the system by `LoadTestAction`.
queues don't need to exist except when actively running load tests (which is These queues don't need to exist except when actively running load tests
not recommended on production environments). There are ten of these queues to (which is not recommended on production environments). There are ten of
provide simple sharding, because the Domain Registry system is capable of these queues to provide simple sharding, because the Domain Registry system
handling significantly more Queries Per Second than the highest throttle limit is capable of handling significantly more Queries Per Second than the
available on task queues (which is 500 qps). highest throttle limit available on task queues (which is 500 qps).
* `lordn-claims` and `lordn-sunrise` -- Pull queues for handling LORDN exports. * `lordn-claims` and `lordn-sunrise` -- Pull queues for handling LORDN
Tasks are enqueued synchronously during EPP commands depending on whether the exports. Tasks are enqueued synchronously during EPP commands depending on
domain name in question has a claims notice ID. whether the domain name in question has a claims notice ID.
* `marksdb` -- Queue for tasks to verify that an upload to NORDN was * `marksdb` -- Queue for tasks to verify that an upload to NORDN was
successfully received and verified. These tasks are enqueued by successfully received and verified. These tasks are enqueued by
`NordnUploadAction` following an upload and are executed by `NordnUploadAction` following an upload and are executed by

12
java/google/registry/env/common/default/WEB-INF/queue.xml vendored
View file

@ -7,18 +7,6 @@
<bucket-size>5</bucket-size> <bucket-size>5</bucket-size>
</queue> </queue>
<queue>
<name>dns-cron</name>
<!-- There is no point allowing more than 10/s because the pull queue that feeds
this job will refuse to service more than 10 qps. See
https://cloud.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/Queue#leaseTasks-long-java.util.concurrent.TimeUnit-long- -->
<rate>10/s</rate>
<bucket-size>100</bucket-size>
<retry-parameters>
<task-retry-limit>1</task-retry-limit>
</retry-parameters>
</queue>
<queue> <queue>
<name>dns-pull</name> <name>dns-pull</name>
<mode>pull</mode> <mode>pull</mode>