zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-06-29 15:53:35 +02:00
Code Issues Projects Releases Packages Wiki Activity

Remove unused dns-cron queue

Browse source 
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=132884314
This commit is contained in:
shikhman 2016-09-12 09:49:42 -07:00 • committed by Ben McIlwain
parent b9b2829f7c
commit cee08d48f2
2 changed files with 72 additions and 83 deletions
Download patch file Download diff file Expand all files Collapse all files

143
g3doc/app-engine-architecture.md
View file

@ -105,77 +105,78 @@ query string parameter "queue" in the url specification for the cron task.
Here are the task queues in use by the system. All are push queues unless Here are the task queues in use by the system. All are push queues unless
explicitly marked as otherwise. explicitly marked as otherwise.
* `bigquery-streaming-metrics` -- Queue for metrics that are asynchronously * `bigquery-streaming-metrics` -- Queue for metrics that are asynchronously
streamed to BigQuery in the `Metrics` class. Tasks are enqueued during EPP streamed to BigQuery in the `Metrics` class. Tasks are enqueued during EPP
flows in `EppController`. This means that there is a lag of a few seconds to flows in `EppController`. This means that there is a lag of a few seconds to
a few minutes between when metrics are generated and when they are queryable a few minutes between when metrics are generated and when they are queryable
in BigQuery, but this is preferable to slowing all EPP flows down and blocking in BigQuery, but this is preferable to slowing all EPP flows down and
them on BigQuery streaming. blocking them on BigQuery streaming.
* `brda` -- Queue for tasks to upload weekly Bulk Registration Data Access * `brda` -- Queue for tasks to upload weekly Bulk Registration Data Access
(BRDA) files to a location where they are available to ICANN. The (BRDA) files to a location where they are available to ICANN. The
`RdeStagingReducer` (part of the RDE MapReduce) creates these tasks at the end `RdeStagingReducer` (part of the RDE MapReduce) creates these tasks at the
of generating an RDE dump. end of generating an RDE dump.
* `delete-commits` -- Cron queue for tasks to regularly delete commit logs that * `delete-commits` -- Cron queue for tasks to regularly delete commit logs
are more than thirty days stale. These tasks execute the that are more than thirty days stale. These tasks execute the
`DeleteOldCommitLogsAction`. `DeleteOldCommitLogsAction`.
* `dns-cron` (cron queue) and `dns-pull` (pull queue) -- A push/pull pair of * `dns-pull` -- A pull queue to enqueue DNS modifications. Cron regularly runs
queues. Cron regularly enqueues tasks in dns-cron each minute, which are then `ReadDnsQueueAction`, which drains the queue, batches modifications by TLD,
executed by `ReadDnsQueueAction`, which leases a batch of tasks from the pull and writes the batches to `dns-publish` to be published to the configured
queue, groups them by TLD, and writes them as a single task to `dns-publish` `DnsWriter` for the TLD.
to be published to the configured DNS writer for the TLD. * `dns-publish` -- Queue for batches of DNS updates to be pushed to DNS
* `dns-publish` -- Queue for batches of DNS updates to be pushed to DNS writers. writers.
* `export-bigquery-poll` -- Queue for tasks to query the success/failure of a * `export-bigquery-poll` -- Queue for tasks to query the success/failure of a
given BigQuery export job. Tasks are enqueued by `BigqueryPollJobAction`. given BigQuery export job. Tasks are enqueued by `BigqueryPollJobAction`.
* `export-commits` -- Queue for tasks to export commit log checkpoints. Tasks * `export-commits` -- Queue for tasks to export commit log checkpoints. Tasks
are enqueued by `CommitLogCheckpointAction` (which is run every minute by are enqueued by `CommitLogCheckpointAction` (which is run every minute by
cron) and executed by `ExportCommitLogDiffAction`. cron) and executed by `ExportCommitLogDiffAction`.
* `export-reserved-terms` -- Cron queue for tasks to export the list of reserved * `export-reserved-terms` -- Cron queue for tasks to export the list of
terms for each TLD. The tasks are executed by `ExportReservedTermsAction`. reserved terms for each TLD. The tasks are executed by
* `export-snapshot` -- Cron and push queue for tasks to load a Datastore `ExportReservedTermsAction`.
snapshot that was stored in Google Cloud Storage and export it to BigQuery. * `export-snapshot` -- Cron and push queue for tasks to load a Datastore
Tasks are enqueued by both cron and `CheckSnapshotServlet` and are executed by snapshot that was stored in Google Cloud Storage and export it to BigQuery.
both `ExportSnapshotServlet` and `LoadSnapshotAction`. Tasks are enqueued by both cron and `CheckSnapshotServlet` and are executed
* `export-snapshot-poll` -- Queue for tasks to check that a Datastore snapshot by both `ExportSnapshotServlet` and `LoadSnapshotAction`.
has been successfully uploaded to Google Cloud Storage (this is an * `export-snapshot-poll` -- Queue for tasks to check that a Datastore snapshot
asynchronous background operation that can take an indeterminate amount of has been successfully uploaded to Google Cloud Storage (this is an
time). Once the snapshot is successfully uploaded, it is imported into asynchronous background operation that can take an indeterminate amount of
BigQuery. Tasks are enqueued by `ExportSnapshotServlet` and executed by time). Once the snapshot is successfully uploaded, it is imported into
`CheckSnapshotServlet`. BigQuery. Tasks are enqueued by `ExportSnapshotServlet` and executed by
* `export-snapshot-update-view` -- Queue for tasks to update the BigQuery views `CheckSnapshotServlet`.
to point to the most recently uploaded snapshot. Tasks are enqueued by * `export-snapshot-update-view` -- Queue for tasks to update the BigQuery
`LoadSnapshotAction` and executed by `UpdateSnapshotViewAction`. views to point to the most recently uploaded snapshot. Tasks are enqueued by
* `flows-async` -- Queue for asynchronous tasks that are enqueued during EPP `LoadSnapshotAction` and executed by `UpdateSnapshotViewAction`.
command flows. Currently all of these tasks correspond to invocations of any * `flows-async` -- Queue for asynchronous tasks that are enqueued during EPP
of the following three MapReduces: `DnsRefreshForHostRenameAction`, command flows. Currently all of these tasks correspond to invocations of any
`DeleteHostResourceAction`, or `DeleteContactResourceAction`. of the following three MapReduces: `DnsRefreshForHostRenameAction`,
* `group-members-sync` -- Cron queue for tasks to sync registrar contacts (not `DeleteHostResourceAction`, or `DeleteContactResourceAction`.
domain contacts!) to Google Groups. Tasks are executed by * `group-members-sync` -- Cron queue for tasks to sync registrar contacts (not
`SyncGroupMembersAction`. domain contacts!) to Google Groups. Tasks are executed by
* `load[0-9]` -- Queues used to load-test the system by `LoadTestAction`. These `SyncGroupMembersAction`.
queues don't need to exist except when actively running load tests (which is * `load[0-9]` -- Queues used to load-test the system by `LoadTestAction`.
not recommended on production environments). There are ten of these queues to These queues don't need to exist except when actively running load tests
provide simple sharding, because the Domain Registry system is capable of (which is not recommended on production environments). There are ten of
handling significantly more Queries Per Second than the highest throttle limit these queues to provide simple sharding, because the Domain Registry system
available on task queues (which is 500 qps). is capable of handling significantly more Queries Per Second than the
* `lordn-claims` and `lordn-sunrise` -- Pull queues for handling LORDN exports. highest throttle limit available on task queues (which is 500 qps).
Tasks are enqueued synchronously during EPP commands depending on whether the * `lordn-claims` and `lordn-sunrise` -- Pull queues for handling LORDN
domain name in question has a claims notice ID. exports. Tasks are enqueued synchronously during EPP commands depending on
* `marksdb` -- Queue for tasks to verify that an upload to NORDN was whether the domain name in question has a claims notice ID.
successfully received and verified. These tasks are enqueued by * `marksdb` -- Queue for tasks to verify that an upload to NORDN was
`NordnUploadAction` following an upload and are executed by successfully received and verified. These tasks are enqueued by
`NordnVerifyAction`. `NordnUploadAction` following an upload and are executed by
* `nordn` -- Cron queue used for NORDN exporting. Tasks are executed by `NordnVerifyAction`.
`NordnUploadAction`, which pulls LORDN data from the `lordn-claims` and * `nordn` -- Cron queue used for NORDN exporting. Tasks are executed by
`lordn-sunrise` pull queues (above). `NordnUploadAction`, which pulls LORDN data from the `lordn-claims` and
* `rde-report` -- Queue for tasks to upload RDE reports to ICANN following `lordn-sunrise` pull queues (above).
successful upload of full RDE files to the escrow provider. Tasks are * `rde-report` -- Queue for tasks to upload RDE reports to ICANN following
enqueued by `RdeUploadAction` and executed by `RdeReportAction`. successful upload of full RDE files to the escrow provider. Tasks are
* `rde-upload` -- Cron queue for tasks to upload already-generated RDE files enqueued by `RdeUploadAction` and executed by `RdeReportAction`.
from Cloud Storage to the escrow provider. Tasks are executed by * `rde-upload` -- Cron queue for tasks to upload already-generated RDE files
`RdeUploadAction`. from Cloud Storage to the escrow provider. Tasks are executed by
* `sheet` -- Queue for tasks to sync registrar updates to a Google Sheets `RdeUploadAction`.
spreadsheet. Tasks are enqueued by `RegistrarServlet` when changes are made * `sheet` -- Queue for tasks to sync registrar updates to a Google Sheets
to registrar fields and are executed by `SyncRegistrarsSheetAction`. spreadsheet. Tasks are enqueued by `RegistrarServlet` when changes are made
to registrar fields and are executed by `SyncRegistrarsSheetAction`.
## Environments ## Environments

12
java/google/registry/env/common/default/WEB-INF/queue.xml vendored
View file

@ -7,18 +7,6 @@
<bucket-size>5</bucket-size> <bucket-size>5</bucket-size>
</queue> </queue>
<queue>
<name>dns-cron</name>
<!-- There is no point allowing more than 10/s because the pull queue that feeds
this job will refuse to service more than 10 qps. See
https://cloud.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/Queue#leaseTasks-long-java.util.concurrent.TimeUnit-long- -->
<rate>10/s</rate>
<bucket-size>100</bucket-size>
<retry-parameters>
<task-retry-limit>1</task-retry-limit>
</retry-parameters>
</queue>
<queue> <queue>
<name>dns-pull</name> <name>dns-pull</name>
<mode>pull</mode> <mode>pull</mode>