Choosing the right catch-up policy for missed cron jobs

A server rebooting for kernel patches at 01:55 and returning at 02:10 caused a critical backup job scheduled for 02:00 to vanish without a trace. Because standard cron daemons do not look back in time, the system silently skipped the execution, leaving the organization without backups for a full day. This incident highlights a common but often overlooked failure mode in self-hosted infrastructure where the default behavior of scheduling tools does not align with business continuity requirements.

What happened

The core issue stems from how different scheduling systems interpret time gaps caused by downtime. In the reported incident, the classic cron daemon checked for jobs matching the current minute. Since the machine was offline at 02:00, that specific minute never existed from the daemon's perspective. Consequently, no error was logged, no alert was fired, and the next successful run did not occur until the following day. This "silent skip" behavior is the default for many traditional Unix scheduling tools, making it dangerous for critical tasks like database dumps or security scans.

In contrast, another team experienced the opposite extreme. Their scheduler was down for six hours, and upon recovery, it attempted to "catch up" by firing 72 instances of a five-minute synchronization job simultaneously. This burst of activity hammered a partner API, triggering rate limits that disrupted service for the rest of the day. Both teams suffered because they had not explicitly defined a missed-run policy for their jobs. The scheduler’s default behavior, rather than a deliberate engineering choice, dictated the outcome.

Key details

• Three main policies: When a scheduled time is missed, systems generally either skip the run, execute it once late, or attempt to run every missed slot.
• Classic cron: Defaults to skipping missed runs entirely, offering no catch-up mechanism or logging for the missed event.
• systemd timers: Using Persistent=true allows a timer to fire once shortly after boot if a run was missed, ideal for daily backups.
• Kubernetes CronJobs: Can be configured with startingDeadlineSeconds to allow late starts within a window, and concurrencyPolicy: Forbid to prevent overlapping executions.
• Per-period jobs: Tasks like billing require processing every missed interval in sequence, which should be handled by the job logic itself, not the scheduler.
• Detection is critical: Regardless of the policy, teams must implement monitoring to detect when a job has been skipped or run significantly late.

Background

To understand these failure modes, it helps to distinguish between stateless syncs and stateful operations. A stateless sync, such as fetching public weather data, usually does not need to recover past data; the next run will simply fetch the current state. For these, skipping is often acceptable. However, stateful operations like backups or financial billing accumulate debt when they miss a cycle. A missed backup means data is unprotected for longer, and a missed billing cycle means revenue leakage.

Traditional cron is designed for simplicity, not resilience. It assumes the machine is always on. Modern alternatives like systemd timers and anacron were built to handle machines that sleep or reboot frequently. Anacron, for example, tracks the last execution time on disk and runs jobs if their period (daily, weekly) has elapsed since the last run. Understanding which tool matches your workload’s reliability needs is essential for robust infrastructure design.

Why it matters

For teams running their own software, the default behavior of your scheduler is a hidden risk. If you rely on standard cron for daily database backups, a simple reboot during maintenance windows can leave you vulnerable for 24 hours without any alert. This silence is particularly dangerous because it creates a false sense of security. Operations teams may see green lights on their dashboards because the cron daemon is running, unaware that the actual work payload was skipped.

Conversely, aggressive catch-up mechanisms can cause self-inflicted denial of service. As seen in the API rate-limiting example, blindly replaying missed jobs can overwhelm downstream services or exhaust local resources. This is especially relevant for microservices architectures where one service’s backlog can cascade into failures across the entire system. Explicitly defining how each job handles downtime prevents these surprises and ensures that recovery actions are proportional to the task’s importance.

What you can do

• Audit your cron jobs: Identify which jobs are critical (backups, billing) and which are ephemeral (syncs, logs). Document the desired behavior for each if a run is missed.
• Migrate critical jobs to systemd timers: Use Persistent=true for daily or weekly tasks to ensure they run once after a reboot if they were missed.
• Configure Kubernetes deadlines: Set startingDeadlineSeconds on CronJobs to define a valid window for late execution, preventing indefinite retries or total skips.
• Implement idempotent catch-up logic: For per-period tasks like billing, write the job to calculate missing intervals from a database record rather than relying on the scheduler to trigger multiple runs.
• Add heartbeat monitoring: Use external monitoring tools to alert if a job has not reported success within its expected period plus a grace margin.
• Log scheduled vs. actual times: Ensure your application logs distinguish between when a job was supposed to run and when it actually started to aid in debugging latency issues.