Llama-server Responses API falsely reports truncated outputs as completed

A significant compatibility bug has been identified in llama-server’s OpenAI-compatible /v1/responses endpoint, affecting self-hosted large language model deployments. As of October 20, 2026, the issue persists in llama.cpp build b11753, where the server incorrectly reports truncated responses and incomplete tool calls as successfully completed. This discrepancy poses a risk for developers building autonomous agents that rely on accurate status signals to execute downstream tasks.

What happened

The core issue lies in how llama-server translates internal generation states into the Responses API format. When a request hits the token limit defined by max_output_tokens, the underlying chat completion engine correctly identifies the stop reason as "length." However, when this result is converted into the Responses API object, the server hard-codes the status field to "completed" regardless of whether the output was finished or cut short. This behavior affects both standard text generation and function calling scenarios.

In testing with models like Gemma-3-1b-it and Qwen3-0.6B, requests constrained to eight tokens returned a status of "completed" with null incomplete details, even though the text ended mid-sentence. More critically, tool calls truncated by token limits returned invalid JSON arguments—such as {"path": "/—yet were still marked as completed items. Clients trusting this status flag may attempt to parse broken JSON or execute partial commands, leading to silent failures or unexpected errors in production environments.

Key details

• The bug affects llama.cpp build b11753 and specifically impacts the /v1/responses endpoint, while /v1/chat/completions correctly reports finish_reason: "length".
• Truncated tool call arguments are returned as valid function_call items with a "completed" status, despite containing malformed JSON that cannot be parsed.
• Streaming responses also fail to emit an response.incomplete event, ending instead with response.completed and omitting the expected function_call_arguments.done signal.
• The only reliable indicator of truncation in the Responses API is checking if usage.output_tokens equals the requested max_output_tokens, though this is a heuristic rather than a definitive status.
• The source code in tools/server/server-task.cpp contains seven instances where the status is hard-coded to "completed," with no logic to read the underlying stop reason or set the status to "incomplete."
• No upstream fix or pull request had been merged for issue ggml-org/llama.cpp#30773 as of October 20, 2026.

Background

The Responses API is a newer specification designed for agent-like interactions, allowing models to return structured outputs including reasoning steps and tool calls. Llama-server implements this by converting incoming Responses requests into standard Chat Completions requests internally. While the generation process works correctly, the translation layer fails to map the chat completion’s finish_reason back to the Responses API’s status and incomplete_details fields. This one-way conversion drops critical failure signals, leaving clients unaware that the model did not finish its thought or command.

For developers, this highlights a common pitfall in compatibility layers: ensuring that error states and boundary conditions are preserved across API translations. A status field is only useful if it can accurately reflect negative outcomes. In this case, the server prioritizes returning content over communicating completeness, assuming that most responses will fit within limits. This assumption breaks down for complex tasks requiring long contexts or precise tool execution.

Why it matters

For teams running self-hosted AI agents, this bug introduces a subtle but dangerous reliability gap. Autonomous systems often rely on the status field to decide whether to proceed with a task. If an agent receives a "completed" status for a truncated file path or code snippet, it may attempt to execute a partial command. This can lead to data corruption, failed deployments, or security vulnerabilities if the partial input is misinterpreted by downstream systems. The risk is highest for small models or complex prompts where token limits are frequently reached.

Furthermore, this issue complicates debugging. Because the server does not report an error, logs may show successful completions even when the output is unusable. Developers might mistakenly attribute the failure to the model’s reasoning capabilities or JSON formatting skills, rather than identifying the infrastructure-level truncation. This misdirection wastes engineering time and can lead to unnecessary model switching or prompt engineering tweaks that do not address the root cause.

What you can do

• Implement client-side validation to treat any response where usage.output_tokens equals max_output_tokens as potentially truncated and verify the output integrity before use.
• Strictly validate all tool call arguments as JSON before execution; reject any function_call item that fails parsing, regardless of its reported status.
• Prefer using the /v1/chat/completions endpoint for critical tasks if your application architecture allows, as it correctly reports finish_reason: "length" for truncated outputs.
• Increase max_output_tokens limits for agent workflows to reduce the frequency of truncation, ensuring the model has sufficient space to complete complex reasoning or tool definitions.
• Run diagnostic scripts like check-llama-responses-status.sh against your deployment to confirm whether your specific version and configuration are affected by this hard-coded status behavior.
• Monitor upstream repositories for patches to llama.cpp issue #30773, and consider contributing a fix that maps the chat completion stop reason to the Responses API status field if you have the capacity.