How a Python version mismatch hid a stale data alarm in cron

On October 9 and 10, 2026, a scheduled telemetry snapshot job triggered an alert claiming a loss of raw archive data. The error message cited an invalid ISO-8601 timestamp, but the underlying issue was not data loss. Instead, a version mismatch in the Python interpreter masked a genuine warning about stale database records, exposing how silent failures can hide behind loud but incorrect error messages.

What happened

The alert originated from a source-freshness check within an agent telemetry pipeline. The job failed with the message F3-SOURCE-CHECK-ERROR: argument --since: not an ISO-8601 timestamp: '1970-01-01T00:00:00Z'. This specific error pointed to a problem with parsing timestamps, leading the operator to initially suspect corruption or loss of the raw archive files. However, the archive was intact, with the newest file dated from that same morning. The failure was isolated to the internal logic checking data freshness, not the data itself.

The root cause lay in the Python version used by the cron scheduler. While the developer’s interactive shell used Python 3.13, the cron job resolved python3 to /usr/bin/python3, which was version 3.9.6. In Python versions prior to 3.11, the datetime.fromisoformat method does not accept timestamps ending with a trailing Z to indicate UTC. The job passed arguments like 1970-01-01T00:00:00Z, causing the parser to crash. This same limitation meant that 540 of the 541 ended_at values in the database, which also ended in Z, could not be parsed correctly by the running interpreter.

Once the parsing issue was resolved by adding a wrapper function to convert trailing Z characters to +00:00, the true nature of the alert emerged. The system was not losing data; it was correctly identifying that the database lagged behind the source files. The newest session in the database had ended on October 1, while newer record files existed from subsequent work sessions. The parse error had been suppressing a valid F3-SOURCE-STALE alert that indicated the database was nearly nine days behind the latest activity.

Key details

• The cron job used Python 3.9.6, which rejects ISO-8601 timestamps with a trailing Z, unlike the developer’s local Python 3.13 environment.
• A custom _fromisoformat() wrapper was implemented to rewrite trailing Z or z to +00:00 before parsing, fixing both CLI and database reads.
• Tests were updated to run under the specific cron interpreter path (PAMSL_PY) to ensure environment parity, reproducing the error before the fix.
• The initial parse error masked a legitimate staleness alert showing the database was 771,145 seconds (about nine days) behind the newest source files.
• Direct database queries using sqlite3 with mode=ro inadvertently created WAL sidecar files, causing subsequent checks to report DB_UNAVAILABLE; switching to immutable=1 resolved this.
• The staleness logic was refined to measure unreflected_wait_seconds for new files rather than just the age of the newest loaded record, preventing false alarms during quiet periods.

Background

Self-hosted automation often relies on cron jobs or similar schedulers that operate in minimal environments. These environments may have different PATH variables, installed packages, or language versions compared to a developer’s interactive shell. When code behaves differently in these two contexts, bugs can remain hidden until they cause production failures. Additionally, monitoring logic must distinguish between temporary gaps in activity and actual system failures. A naive check might flag a system as stale simply because no new data arrived over a weekend, whereas a robust check accounts for expected batch processing windows and grace periods.

SQLite databases operating in Write-Ahead Logging (WAL) mode create additional files (-wal and -shm) to manage concurrent access and durability. Read-only connections that do not explicitly declare immutability can still trigger the creation of these sidecar files, potentially interfering with other processes that monitor file presence as a lock mechanism. Understanding these low-level behaviors is critical for maintaining reliable self-hosted services where direct database access is part of the operational workflow.

Why it matters

For teams running their own software, environment consistency is a frequent source of subtle bugs. Code that passes all tests in a modern development environment may fail in production if the runtime version differs. This incident highlights the importance of testing against the exact interpreter and environment variables used by the scheduler. Without this parity, errors like the fromisoformat rejection can go undetected until they mask more significant operational issues, such as data staleness.

Furthermore, monitoring logic requires careful tuning to avoid alert fatigue while ensuring real problems are caught. The original staleness check flagged the system based on the age of the newest loaded record, which could be misleading after a quiet period. By shifting the metric to measure how long new files have waited to be ingested, the team created a more accurate signal. This approach distinguishes between a system that is legitimately waiting for its next scheduled build and one that has fallen behind due to an error.

What you can do

• Verify the Python version used by your cron jobs matches your development environment, or add compatibility layers for known differences like ISO-8601 parsing.
• Run integration tests using the exact interpreter path and environment variables configured in your production scheduler to catch version-specific bugs.
• Implement wrapper functions for standard library calls that have known version-dependent behaviors, ensuring consistent parsing across environments.
• Refine staleness checks to measure the wait time of unprocessed data rather than just the age of the last processed item, reducing false positives during quiet periods.
• Use immutable=1 in SQLite connection strings for read-only diagnostic queries to prevent the creation of WAL sidecar files that might interfere with other processes.
• Ensure monitoring alerts provide clear, actionable context, and regularly review alert logic to confirm it reflects current operational patterns and batch schedules.