Commit graph

25 commits

Author SHA1 Message Date
mcilwain
1d134cdd3f Delete the verify entity integrity mapreduce
We never really used it and it'll be obsolete come Registry 3.0 anyway.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=213274520
2018-09-20 11:19:36 -04:00
weiminyu
e72f5c09a2 Enable Premium terms export in production
Defines cron job in crash, sandbox and production environments.
Job already exists in alpha.

Job is not added to qa environment.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=212878436
2018-09-14 11:56:42 -04:00
mmuller
e4bb1c281c Uncomment crontab entry for deleteProberData
This was only supposed to stay commented out until load-testing was complete.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=210087917
2018-09-08 00:04:26 -04:00
guyben
11c5d11a29 Re-enable the RdeUpload cronjob
RdeUpload works on alpha and sandbox by sending the data to a google internal
server. It is still running (and succeeding) after each RdeStaging, but the
lack of the cron job means it is currently lagging behind.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=202181148
2018-06-27 15:28:53 -04:00
guyben
9d2b1e7572 Consolidate all Set parameter parsing
Currently, we have two different ways to parse a "set" parameter:
key=value1&key=value2&key=value3...
and
keys=value1,value2,value3

This is error prone for several reasons:
- different parts of the code must be "synchronized" to use the same style (the
  place that creates the request, and the place that parses the request)
- for the key=value1&key=value2, we often use the same key name for the single
  value and the set value. This can result in subtle bugs where part of the
  code will successfully read the key assuming there's only one key (and will
  get the first key=value1, ignoring the rest)

Here we transition everything to the keys=value1,value2,value3 method. This one
was chosen because:
- it's shorter
- it's more intuitive for users
- the key name is plural, differentiating it from the singular key=value that
  other requests might need

-----------------------------------

To make sure there are not "transition issues", we will continue to support
(with warnings) the key=value1&key=value2 parameter parsing until we're sure we
haven't forgotten to update any part of the code.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=198810681
2018-06-06 15:04:02 -04:00
mcilwain
8f1848e32e Disable verify entity integrity mapreduce on sandbox
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192289233
2018-04-10 16:59:28 -04:00
mcilwain
a8b6195ce2 Make RDE run less frequently on sandbox/alpha
This also removes RDE tasks that shouldn't/can't run on non-production environments, like upload/reporting.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192177779
2018-04-10 16:56:22 -04:00
mcilwain
e816913c61 Increase # of commit log buckets ~4X for all non-prod environments
This also reduces the interval of the commitLogCheckpoint cron job to once
every three minutes, as this job needs to load all commit log bucket entities.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=191613858
2018-04-10 16:33:47 -04:00
guyben
63785e5149 Remove empty TLD parameter when fanning out without TLDs
TldFanoutAction fans out a given endpoint to all TLDs (either TEST, REAL, or
both).

However, it is also used to delegate a single endpoint request that we want set
in a specific queue (so we can control retries). We do that by setting the TLD
list to "runInEmpty" rather than "forEachRealTld" or "forEachTestTld".

Currently, using "runInEmpty" would still specify a TLD - but that TLD would be
the empty string. This is a bug: it sets the TLD parameter to a bad value. It
worked only because none of the endpoints called with "runInEmpty" were using
the TLD parameter.

However, this will (and does) break if either (a) the endpoint accepts an
optional TLD parameter (like deleteProberData does), or (b) the given endpoint
already has a TLD parameter in it (we want to run the endpoint with a single
TLD, but still use the "fanout" to set the right queue).

This CL fixes several things:

- if runInEmpty is given, no TLD parameter is added
- 'runInEmpty' is now mutually exclusive with 'forEach*Tld' and 'excludes'
- we do some sanity checks and added logging
- removed the buggy and unused "':tld' in path is replaced by TLD"
- in the cron.xml, removed documentation for :tld and the broken :registrar

Note that none of the endpoints that were used with runInEmpty fanout had the TLD parameter prior to deleteProberData

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189954585
2018-04-02 16:24:27 -04:00
guyben
27894df45f Turn off deleteProberData on alpha for duration of loadtesting
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=187484858
2018-03-06 19:09:52 -05:00
guyben
8beb10c2a3 Update sandbox / alpha cron.xml to be in line with production
There are 2 types of changed done here:
- reorder the existing cron jobs to be in the same order as production (for
  easier diffing)
- add missing cron-jobs to either alpha or sandbox

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=183232936
2018-02-01 21:57:39 -05:00
guyben
0e3d050dae Temporarily disable deleteProberData cron job in sandbox for load-testing
Loadtesting data is identified as "prober data" by this job (it removes
anything under ".test", not only prober data)

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177309096
2017-12-01 22:14:06 -05:00
mcilwain
7dc1940cdb Move ResaveAllEppResources mapreduce from tools service to backend
It makes sense for all mapreduces to run in backend, especially onces
that are scheduled regularly to run in cron like this one now. We don't
have many instances configured for the tools service anymore on some
of our environments, so backend is the friendliest place for a mapreduce
to run.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168882122
2017-09-20 10:27:17 -04:00
guyben
d7214b58fc Re-enable DeleteOldCommitLogs cron job
Also adds a "resave all epp" cron job that's needed for the delete to work correctly.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=168879965
2017-09-20 10:27:17 -04:00
mcilwain
cdadb54acd Refer to Datastore everywhere correctly by its capitalized form
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=147479683
2017-02-17 12:12:12 -05:00
shikhman
d3fe6be385 Remove ECatcher from GAE code
It's superceded by Stackdriver Trace.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=145440930
2017-01-25 12:25:41 -05:00
ctingue
f30b87d40e Add ExpandRecurringBillingEventsAction cron job
Note that this merely starts this MR on a daily schedule -- the billing queries that ultimately consume the synthetic OneTime events are filtering out the events at this time, so we're still relying on query-time expansion of Recurrings.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=144450565
2017-01-18 11:03:32 -05:00
mountford
890354938c Switch TMCH SMDRL fetch job to start at 00:15 and 12:15
The job was starting at midnight and noon, which is exactly when the files are changing. This resulted in intermittent failures, as the files are temporarily missing during the changeover.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=139081163
2016-11-15 11:02:42 -05:00
mountford
5f691d6329 Enable built-in session cleanup in alpha and sandbox
App Engine provides a servlet which deletes up to 100 expired _ah_SESSION entities from DataStore. This CL adds a cron job to call the servlet every 15 minutes in both alpha and sandbox. Assuming all goes well, we will turn it on in production.

I originally learned about this servlet here:

http://www.radomirml.com/blog/2011/03/26/cleaning-up-expired-sessions-from-app-engine-datastore/

But it appears that we do not need a servlet definition, just a cron entry.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=137533532
2016-11-02 15:19:34 -04:00
mcilwain
6636e02d57 Clean up and consolidate some unused queues
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=136170276
2016-10-14 17:37:21 -04:00
mcilwain
e192c11adf Add a batched [] for DNS refreshing on host renames
This will replace the existing DnsRefreshForHostRenameAction.

This is stage one of a three stage migration process. It adds the new queue and
[] but doesn't call them yet. Stage two will cut over to using the new
functionality, and stage three will remove the old functionality.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134793963
2016-10-03 16:11:00 -04:00
mcilwain
7f0cb4eae5 Move prober deletion action to backend module
Also creates a new package named 'batch' to house it.

TESTED=I deployed it to alpha, sent a POST request to the task URL, and it
successfully ran the [].

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134332999
2016-10-03 15:41:38 -04:00
mcilwain
00ea99960a Improve efficiency of async contact and host deletion with batching
This allows handling of N asynchronous deletion requests simultaneously instead
of just 1.  An accumulation pull queue is used for deletion requests, and the
async deletion [] is now fired off whenever that pull queue isn't empty,
and processes many tasks at once.  This doesn't particularly take more time,
because the bulk of the cost of the async delete operation is simply iterating
over all DomainBases (which has to happen regardless of how many contacts and
hosts are being deleted).

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=133169336
2016-09-19 11:47:55 -04:00
mcilwain
c06133435b Consolidate some cron groups and use fanout action for more retrying
This is the first step in consolidating our task queues down into a smaller
number.  We have lots of tasks that run quite infrequently, and they can all run
in the same queue to get retry semantics without needing a new queue for each
cron entry.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=131990472
2016-09-02 13:56:23 -04:00
Justine Tunney
7cc7dc4af2 Use zip_rule() for EAR/WAR deploy archives
These build rules allow Bazel to generate the .ear and .war files which
appcfg.sh (a tool that comes with the App Engine SDK) can then use to
perform a deployment.

Included in this CL are configurations for five separate production
environments: production, sandbox, alpha, crash, and local.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=129163010
2016-08-02 21:00:39 -04:00