Facet engineers fix silent data loss bug in multi-tenant SQLite backup system

Facet, a creator platform built on Go and SQLite, recently identified and resolved a critical bug in its point-in-time restore mechanism. The issue, discovered in October 2026, involved the system incorrectly selecting outdated database snapshots during recovery operations. Engineers implemented a three-part fix to ensure restores rely on timestamps rather than simple sequence counters, preventing silent data regression for tenants.

What happened

Facet operates as a multi-tenant platform where each creator receives an isolated SQLite database. To provide granular recovery options without prohibitive costs, the team built a custom backup shipper integrated directly into their application process. This system uploads immutable snapshot and Write-Ahead Log (WAL) segments to Cloudflare R2 storage. Each object includes a signed header with a sequence number and a hash of the previous object, creating a verifiable chain of custody for every tenant’s data.

The engineering team recently detected that several demo sites were pointing to stale data chains after being migrated from a staging environment to production. The staging hosts had maintained local state files with high epoch numbers, while the new production instances started fresh at epoch one. Because the restore logic prioritized the highest epoch number regardless of when the data was written, the system would have restored sites to their state from late September instead of the current live version. This occurred despite all individual backup objects passing integrity checks, as the error lay in the selection logic rather than data corruption.

To resolve this, the team deployed three specific changes to the production environment. First, they modified epoch allocation to check remote history before assigning new sequence numbers, preventing local state files from creating conflicting chains. Second, the restore planner now selects epochs based on temporal relevance to the target restore time, not just numerical height. Finally, the system now explicitly refuses to restore if it detects inverted chronology, forcing operators to address ambiguous timelines rather than guessing.

Key details

• The backup system costs approximately $0.0034 per tenant per month by using change-driven uploads that skip idle databases.
• Each backup object is immutable and contains a SHA-256 hash of the previous object, allowing verification without a central index.
• The bug caused the restore planner to select stale chains with higher epoch numbers from previous staging environments.
• Fixes include allocating epochs against remote history and refusing restores when chronology is inverted.
• Post-fix testing showed a demo site with 79 WAL segments restored successfully in 2 minutes and 34 seconds.
• The system disables automatic SQLite checkpointing for shipped tenants to prevent write conflicts with the internal shipper.

Background

Point-in-time restore (PITR) allows administrators to recover a database to any specific moment, rather than just the last full backup. In SQLite, this typically involves combining a base snapshot with a series of WAL segments that record every change made since that snapshot. Facet’s approach avoids external tools like Litestream by running the backup logic within the same Go process that serves the database. This ensures the backup system is aware of tenant lifecycle events such as hibernation or migration, which external sidecars might miss.

The concept of an "epoch" in this context refers to a distinct period in the backup chain. Epochs reset when continuity cannot be guaranteed, such as after a host migration. By chaining these epochs with cryptographic hashes, the system ensures that any gap or tampering is detectable. However, managing these epochs across different environments requires careful coordination to ensure that newer numbers always correspond to newer data, a assumption that failed in the recent incident.

Why it matters

For teams running self-hosted software or managing multi-tenant architectures, this incident highlights the risks of relying on local state for distributed systems. When applications move between hosts or environments, leftover metadata can create conflicts that are difficult to detect until a restore is attempted. The Facet team’s experience shows that even robust integrity checks on individual files cannot prevent logical errors in how those files are assembled during recovery.

The cost efficiency of their model also offers a blueprint for small and mid-sized companies. By charging only for actual write operations and skipping idle tenants, they achieved a price point that makes PITR viable for every user, not just enterprise clients. This demonstrates that granular backup strategies do not necessarily require expensive infrastructure if the software architecture aligns with the storage provider’s pricing model.

Furthermore, the decision to fail loudly when encountering ambiguous data is a crucial lesson in reliability engineering. Many systems might attempt to guess the correct chain, potentially serving corrupted or outdated data without alerting the user. Facet’s choice to block the restore and require manual intervention ensures that data integrity is never compromised for the sake of availability.

What you can do

• Audit your backup selection logic to ensure it prioritizes timestamps over sequence numbers when resolving conflicts.
• Implement change-driven upload mechanisms to reduce storage costs for idle databases in multi-tenant setups.
• Verify backup integrity by comparing restored table row counts against live databases during regular drills.
• Ensure backup processes are aware of tenant lifecycle events like migrations to prevent state desynchronization.
• Configure systems to refuse ambiguous restores rather than attempting to guess the correct data chain.
• Store backup metadata remotely rather than relying solely on local state files that may become stale after moves.