Why measuring one reader led to deleting files needed by tests

On October 31, 2026, a developer working on a content-collection system moved 34,557 HTML files to quarantine after determining that a specific reader process never accessed them. The immediate collection round completed successfully, but a subsequent comparison against a sealed test baseline revealed 310 new failures. The files were restored the same day, avoiding permanent data loss, but the incident exposed significant gaps in how the system tracked file usage across different components.

What happened

The system’s report directory had grown to 132.8GiB without triggering any alarms, as jobs running every 15 minutes created new folders without cleanup. An initial reference-based pruning pass reduced the size to 115.1GiB by removing unreferenced units and compacting a SQLite database, but most remaining data was still referenced by the collection state. To address the remaining bulk, the developer measured which files a collection-state validator opened during a run from 16:50 to 17:40 KST. Of 46,643 collected HTML files totaling 52.4GiB, the validator only opened 1,920 files. The remaining 34,557 files, mostly history entries for re-fetched videos, were deemed unnecessary and moved to quarantine.

However, when the full test suite was executed, it produced 310 failures beyond the existing baseline. These errors included ValueError exceptions regarding unavailable YouTube page bodies and FileNotFoundError issues for missing HTML files. The failures occurred because other processes, such as publish replay and backlog judging, still required access to these raw capture bodies. The developer had mistakenly assumed that because one reader did not open the files, no part of the system needed them. The quarantine strategy allowed for a quick recovery by renaming the files back, restoring the system to a passing state with zero missing files.

Key details

• Initial measurement error: The developer measured file access for only one reader (the collection-state validator), ignoring other components like publication queue replays.
• Scale of impact: 34,557 HTML files totaling 37.8GiB were moved to quarantine based on incomplete data.
• Test suite failure: Running the full test suite revealed 310 new failures, primarily due to missing raw HTML files needed for replay and provenance tracking.
• Recovery method: Files were moved rather than deleted, allowing restoration via rename without requiring a re-crawl of external sources.
• Recorder traps: The file-access recorder missed reads due to relative path mismatches in Python 3.13.0 audit hooks and lost records when the main pytest process exited via os._exit(1).
• Current status: As of November 4, 2026, the storage gate is red because three of four nightly runs ended in a restored state, preventing the final deletion step from executing.

Background

In complex software systems, especially those handling large volumes of data like content pipelines, understanding which files are actively used is critical for maintenance. A "reader" in this context refers to any process or component that opens and processes data files. Developers often use monitoring tools or audit hooks to track file access, creating a "read set" of files that were opened during a specific operation. The complement of this set—files that were not opened—is sometimes assumed to be safe for deletion. However, this assumption is dangerous if the monitoring does not cover all possible readers, including background jobs, test suites, or rare edge-case handlers.

Quarantining is a safety practice where files are moved to a separate location rather than being permanently deleted. This allows teams to verify that the system functions correctly without the data before committing to irreversible removal. If issues arise, the files can be moved back quickly. This approach contrasts with direct deletion, which would require expensive and time-consuming re-fetching of data from external sources if the decision proves incorrect.

Why it matters

For teams running self-hosted software or managing internal data pipelines, this incident illustrates the hidden complexity of data lifecycle management. It is easy to assume that static analysis or limited runtime monitoring provides a complete picture of data dependencies. In reality, systems often have latent dependencies where data is only accessed during specific events, such as error recovery, historical re-processing, or comprehensive testing. Deleting data based on partial visibility can lead to subtle failures that are difficult to diagnose and expensive to repair.

The reliance on a single metric—the read set of one validator—created a false sense of security. The fact that the immediate collection round passed after the quarantine misled the developer into thinking the change was safe. It was only the broader test suite, which simulates various operational scenarios including replays and historical joins, that revealed the missing dependencies. This highlights the importance of using comprehensive test suites as a validation tool for infrastructure changes, not just code logic.

Furthermore, the technical pitfalls encountered with the file-access recorder demonstrate that even monitoring tools can have blind spots. Issues like relative path resolution in audit hooks or process exit behaviors can lead to undercounting file accesses. In data cleanup operations, an undercount is particularly dangerous because it directly increases the risk of deleting active data. Teams must validate their monitoring tools against known baselines to ensure they are capturing all relevant activity.

What you can do

• Move before deleting: Always move candidate files to a quarantine directory with a fixed timer before permanent deletion, ensuring that mistakes can be reversed cheaply.
• Measure all readers: Compute delete candidates against every component that might access the data, including background jobs, replay mechanisms, and the full test suite.
• Validate monitoring tools: Test your file-access recorders against readers with known behavior to ensure they are not missing reads due to path resolution or process exit issues.
• Run full test suites: After any data movement, compare the results of the full test suite against a sealed baseline to catch latent dependencies that immediate smoke tests might miss.
• Implement tripwires: Set up jobs that monitor logs for attempts to read missing files, allowing you to detect and restore accidentally deleted data before it causes widespread failure.
• Exclude quarantine from budgets: When calculating storage usage, exclude quarantined bytes from the main budget to avoid confusion about actual active data size while keeping them available for recovery.