Green backup logs do not prove data recovery works
A developer demonstrates that successful cron jobs often mask broken backups, showing how automated restore drills verify actual data integrity.
A developer recently discovered that their nightly database backup job had reported success for months without ever being tested for actual data recovery. By simulating data loss and attempting a restore, they revealed that a green status log only confirms a file was written, not that the data inside is usable or complete.
What happened
The author, who works on the self-hosted deployment platform Levelrail, decided to audit their own backup practices after realizing they could not recall the last time they successfully loaded a backup file into a database. They set up a test environment using Postgres 16 on a Mac with Docker, running Levelrail built from the v1.0.1 release branch. The test involved creating a database with specific row counts and financial totals, scheduling a backup, and then intentionally deleting data from the live system to simulate a catastrophic error.
After triggering the backup and deleting 500 orders from the live database, the author used the restore-as-new command to create a separate database from the backup file. This process left the damaged live database untouched while spinning up a fresh instance from the backup. Comparing the two revealed that the restored database contained the original 2000 orders and correct financial totals, while the live database showed the damage. This manual test proved the backup mechanism worked, but it also highlighted that the daily cron job had never provided this level of assurance.
To automate this verification, the author explored Levelrail’s "drill" feature. Unlike a simple file existence check, a drill downloads the backup, verifies its SHA-256 checksum, restores it into an isolated container, and validates the content. The author noted that drills require an object-store target rather than local storage, prompting them to configure an S3-compatible endpoint. Once configured, the drill passed all checks, including checksum validation and content matching, changing the database health status from unverified to healthy.
Key details
- A successful backup log only proves a file was written, not that the data can be restored or is consistent.
- Manual restore tests should compare row counts and financial sums to detect subtle data corruption.
- The
restore-as-newcommand creates a separate database instance, ensuring the live production data remains unaffected during testing. - Automated restore drills in Levelrail verify checksums and content for Postgres, MySQL, and MariaDB databases.
- Drills require an object-store target like S3 or R2; local storage targets do not support automated drill operations.
- Backup copies to a second destination can be enabled to protect against single-bucket failures, though this was only reviewed in documentation.
Background
In many self-hosted environments, administrators rely on cron jobs to run scripts like pg_dump and save the output to a disk or cloud bucket. These scripts typically exit with a zero status code if the file write succeeds, leading monitoring tools to report a green status. However, this does not account for issues such as corrupted files, incomplete writes, or schema incompatibilities that might prevent a successful restore. Without a regular restore test, teams operate under the false assumption that their disaster recovery plan is functional.
Levelrail is a self-hosted deployment platform designed to simplify infrastructure management without Kubernetes. It includes built-in tools for managing databases and backups, aiming to replace manual scripting with integrated workflows. The platform supports various storage providers, including AWS S3, Cloudflare R2, and custom S3-compatible endpoints, allowing users to store backups off-site for better durability.
Why it matters
For teams running their own software, the difference between a backed-up file and a recoverable database is critical. A green monitoring dashboard can create a dangerous sense of security, leading engineers to believe they are protected when they are not. If a real disaster occurs, such as a ransomware attack or accidental deletion, the first time a team attempts a restore may be the first time they discover the backup is unusable. This delay can extend downtime from minutes to days, causing significant business impact.
Automating the verification process removes the reliance on human memory and discipline. Manual restore tests are often skipped due to time constraints or fear of impacting production systems. By integrating restore drills into the backup workflow, teams can ensure that every backup is not just a file on a disk, but a verified snapshot of their data. This shift from passive logging to active verification transforms backup strategies from hopeful gestures into reliable safety nets.
What you can do
- Schedule a manual restore test immediately using a non-production environment to verify your current backup files.
- Compare row counts and critical aggregate values, such as financial totals, between live and restored databases to ensure data integrity.
- Configure automated restore drills if your platform supports them, ensuring they run on a regular schedule like weekly.
- Use object storage targets for backups instead of local disk storage to enable advanced features like automated drills and off-site redundancy.
- Enable backup copies to a second destination to protect against storage provider failures or accidental deletions.
- Document your restore procedure and include steps for verifying data consistency, not just file existence.



