Silent failure: how a metadata check blocked all inference jobs

A recent technical case study published on November 3, 2026, details a subtle but critical failure mode in a self-hosted AI inference service. The author, Manh Liem, discovered that while the user interface and API endpoints appeared functional, every background job failed silently due to a misconfigured health check. This incident highlights the danger of relying on indirect metrics for system liveness rather than testing the actual execution path.

What happened

The incident began when the author ran a paid red-team assessment against a hosted endpoint. From the outside, the service looked operational. The buyer page loaded correctly, form submissions returned a 202 Accepted status, and the job polling endpoint responded with data. However, every single job eventually returned an error status with the message "model endpoint unreachable." The error pointed to a local address, suggesting a connectivity issue within the infrastructure itself.

Upon investigation, the root cause was not a network outage or a crashed model server. The worker process responsible for executing jobs performed a liveness check before starting any work. This check queried a GET /v1/models route, which was part of a local proxy responsible for managing a roster of hosted providers. Building this roster required a refresh token for a third-party account. That token had expired, causing the roster endpoint to return a 500 Internal Server Error. The liveness logic interpreted any non-200 response as proof that the model was unreachable, so it aborted the job immediately without ever attempting to run the actual inference.

The actual inference route, POST /v1/chat/completions, was fully functional throughout the outage. It was served by a different component, a provider router, which did not depend on the expired token. The author confirmed this by sending a minimal request with max_tokens set to 1, which returned a valid completion. The system had a working production path and a broken metadata path, but the gatekeeper checked the wrong one.

Key details

• The failure was silent because every individual HTTP response was technically correct: the page returned 200, the submission returned 202, and the liveness check returned 500.
• The liveness probe targeted GET /v1/models, a metadata endpoint, instead of the actual inference endpoint POST /v1/chat/completions.
• An expired refresh token for a third-party provider caused the metadata endpoint to fail with a 500 error.
• The worker treated the metadata failure as a total system outage, preventing any real jobs from executing.
• Logs showed no contradictions, making the failure difficult to detect through standard anomaly detection.
• The fix involved changing the probe to test the actual inference route with a minimal payload.

Background

In distributed systems, health checks are used to determine if a service is ready to accept traffic. A liveness probe typically checks if a process is running, while a readiness probe checks if it can handle requests. Ideally, these probes should mimic the actual work the service performs. If a probe checks a dependency that is not critical for the core function, or if it checks a side effect like metadata generation, it can produce false negatives. This creates a situation where the system is capable of doing its job but refuses to start because a peripheral component is unhealthy.

This case also illustrates the difference between component health and chain health. A microservice architecture may have many independent components, such as a web server, a queue, a worker, and a model router. Each component can report "healthy" individually, yet the end-to-end flow can be broken. For example, the web server can accept requests, and the model can generate text, but if the worker fails to bridge them due to a configuration error, the user experience is a total outage. Monitoring individual components is insufficient; teams must verify that the entire chain executes successfully.

Why it matters

For teams that run their own software, this incident serves as a warning against over-relying on standard health check patterns. Many frameworks provide default liveness endpoints that check database connections or basic process status. If your application has complex dependencies, such as external API tokens or dynamic configuration rosters, these defaults may not reflect the true state of the system. A silent failure like this can persist for days, draining resources and frustrating users, without triggering any alerts because the monitoring system sees exactly what it expects to see: consistent errors that match the configured logic.

Furthermore, this highlights the cost of indirect testing. Checking a metadata endpoint is cheaper and faster than running a real job, which is why developers often choose it. However, as shown here, metadata paths can diverge from production paths. They may have different authentication requirements, rate limits, or code paths. When they fail, they do not necessarily indicate that the core product is broken. Relying on them creates a fragile monitoring setup that alerts on irrelevant issues while missing critical ones. Teams must balance the cost of synthetic transactions with the risk of undetected outages.

What you can do

• Audit your health checks to ensure they test the critical path of your application, not just peripheral dependencies.
• Implement synthetic monitoring that runs a minimal end-to-end transaction, such as a single-token inference request, at regular intervals.
• Separate liveness checks (is the process running?) from readiness checks (can it do the work?) and ensure readiness checks mimic real user actions.
• Monitor for business-level metrics, such as job completion rates or empty reports, rather than just HTTP status codes.
• Set up alerts for specific error patterns in logs, even if the overall system status appears green.
• Regularly rotate and test third-party tokens and credentials in a staging environment to prevent expiration surprises in production.