implemented (in a team estesstvenno) large social network, a lot of CZK tasks. Many scripts must be constantly spinning (I guess you could call them demons, but run for the crown).
Had to develop a system of deployment and management tasks/scripts (tasks spinning on different servers).
The basic principles are as follows:
All cron tasks are heirs from the base class.
All cron tasks are run from a single script (wrapper) run.
Crowns each task has a pid file ( in case that does not start at the same time two identical tasks )
If you want to run simultaneously two of the same script, then in this case the pid file had the extension
for example crontask.1.pid crontask.2.pid
Every cron script to a centralized DB ( Soi network organized sharding, all data are divided into shards ) sent data: start time run end time run, how many made (a kind of measure, for example number of processed items in the queue)
there were two script analysis.
the first script monitors the current data in the database, compared them with a template (what should be) and gave Nagios 0 1 2
sysadmin for Nagios if they saw something wrong, then run the monitoring table scripts and observed data on scripts which when and how did
here's a layered cake