How an idempotency guard stalled a publishing daemon for ten hours

A self-hosted publishing daemon stopped processing new content for ten hours on October 23, 2026, due to a subtle mismatch in how success and failure were defined. The incident occurred when the daemon encountered an article that was already live, interpreting the publisher’s idempotency guard as a critical error rather than a completed task.

What happened

The daemon manages a pool of articles in sorted order, processing them one by one. It marks an item as used only after receiving a success signal. When it encountered an article titled "doomsday_escrow," the underlying publisher tool checked if the slug already existed. Finding the article live, the publisher returned exit code 4 with the message "ALREADY PUBLISHED." This response is designed to prevent duplicate posts, acting as a safety mechanism for retries.

However, the daemon was programmed to treat any non-zero exit code as a failure. Because it did not mark the item as used, it remained at the front of the queue. On the next cycle, the daemon picked up the same item, received the same "already published" response, and failed again. This loop repeated indefinitely, preventing the daemon from ever advancing to the subsequent items in the pool. The logs showed a busy worker, but in reality, the queue was permanently stalled behind a single item that had already been successfully processed.

The root cause was a semantic disagreement between the two tools. The publisher viewed exit code 4 as a valid state indicating the work was done, while the daemon viewed it strictly as an error. The developer noted that the publisher also uses exit code 4 for genuine failures, such as "HTML_NOT_ALLOWED," which requires user intervention. This meant the fix could not simply ignore all exit code 4 responses; it had to distinguish between a successful idempotent state and a true content error.

Key details

• The daemon stalled for ten hours, logging repeated failures for the same article.
• The publisher returned exit code 4 and the string "ALREADY PUBLISHED" to indicate the post was already live.
• The daemon interpreted exit code 4 as a generic failure and did not advance the queue pointer.
• Exit code 4 was also used for "HTML_NOT_ALLOWED," a true failure requiring content changes.
• The fix required checking both the return code and the output message to differentiate success from failure.
• The corrected logic returns true if the code is 4 and the message is "ALREADY PUBLISHED," otherwise it checks for code 0 and "VERIFY."

Background

Idempotency is a property of operations where performing the same action multiple times has the same effect as performing it once. In software systems, especially those involving network requests or file writes, idempotency guards prevent duplicate entries or corrupted states. For example, if a payment service charges a card twice due to a network timeout, an idempotency key ensures the second request is recognized as a duplicate and ignored.

In this case, the publisher used an idempotency guard to ensure that retrying a publish command would not create a duplicate article with a suffix like "-1." Instead of returning a standard success code (0), it returned a specific code (4) to signal that the desired end state already existed. Systems calling such tools must understand that "already done" is a form of success, even if the technical return value looks like an error code.

Why it matters

For teams running their own software, particularly those relying on cron jobs or background daemons, this incident highlights the fragility of queue-based systems. A single misinterpreted status code can halt an entire workflow, leading to data delays or missed SLAs. Unlike cloud services that often have built-in retry logic and dead-letter queues, self-hosted daemons may lack these safeguards, making them susceptible to silent stalls.

The visibility of such failures is also critical. The developer noted that ten hours of identical log lines appeared normal at a glance, masking the fact that no progress was being made. Without explicit alerts for stalled queues or repeated failures on the same item, operators may assume the system is healthy while it is actually stuck. This underscores the need for monitoring that tracks progress, not just activity.

Furthermore, this case illustrates the importance of precise error handling. Relying solely on exit codes without inspecting output messages can lead to incorrect assumptions about system state. When tools use non-standard codes for specific conditions, callers must implement logic that respects the semantic meaning of those codes, not just their numerical value.

What you can do

• Review your daemon logs for repeated failures on the same item, which may indicate a stalled queue rather than transient errors.
• Check if any external tools you call use non-zero exit codes for idempotent successes, such as "already exists" or "no change needed."
• Update your error handling logic to inspect output messages alongside exit codes, ensuring you distinguish between true failures and idempotent states.
• Implement alerts for jobs that fail repeatedly with the same error, signaling a potential blockage in the workflow.
• Test your queue processing by manually running failed subprocesses to verify if their postconditions are already met.
• Add logging that explicitly states when a queue is stalled due to a recurring item, making the issue visible immediately.