Local LLM benchmark reveals extraction flaws in knowledge graph building

A new benchmark published on November 6, 2026, evaluates how well local large language models build knowledge graphs from text. The study, conducted by Denis MacPherson for Chaos Cypher, tests 22 models available via Ollama to determine which ones can accurately extract entities and relationships without hallucinating facts.

What happened

The benchmark separates the task of building a knowledge graph into two distinct phases: extraction and chat. In the extraction phase, models must read passages, identify specific entities and relationships, and cite the exact sentence that supports each claim. This is done using strict pure-function checks rather than another AI judge, ensuring objective pass or fail results. The test includes 65 isolated probes and 39 probes embedded within larger chunks of text from Leo Tolstoy’s War and Peace to simulate real-world document processing.

In the chat phase, models answer questions based solely on the previously extracted graph. They must provide correct answers, avoid inventing information not present in the context, and explicitly decline to answer if the data is missing. The final overall score weights extraction at 60 percent and chat at 40 percent, with extra emphasis on the "in-chunks" extraction performance since this mirrors how models handle actual documents.

The results highlight a significant divergence between model sizes and their actual performance. Larger models do not automatically perform better; for instance, Gemma 4 12B performed competitively against much larger models, while several 30B-class models struggled with the structured extraction tasks. The study emphasizes that adherence to strict output formats and evidence discipline is a training characteristic rather than a function of parameter count.

Key details

• The benchmark evaluates 22 local models from Ollama using a reproducible command-line tool called chaoscypher-cli.
• Extraction scores vary widely among models, while chat scores are clustered closely together, making extraction the primary differentiator.
• Gemma 4 31B achieved the highest overall score with no flags, serving as the reference model for the chat column tests.
• Models like GPT-OSS 20B and DeepSeek-R1 14B were flagged for ignoring "thinking-off" settings, causing them to waste output tokens on internal reasoning.
• Every score includes a 90% confidence interval derived from resampling tasks 2,000 times, indicating that many top-ranked models are statistically tied.
• The tool allows users to run benchmarks locally with commands like chaoscypher benchmark run probes --local-only to validate models on their own hardware.

Background

Knowledge graph extraction requires a model to act as a structured data parser rather than a creative writer. Unlike standard chat interactions, where fluency is prized, extraction demands rigid adherence to format, precise entity naming, and strict evidence citation. If a model fails to cite the correct sentence or invents a relationship, the resulting graph becomes unreliable for downstream applications.

Retrieval-Augmented Generation (RAG) systems often rely on these graphs to answer user questions. If the underlying graph is messy due to poor extraction, the chat component will inevitably produce incorrect or hallucinated answers, regardless of how sophisticated the chat model itself is. This benchmark isolates these two capabilities to help developers choose the right model for each stage of their pipeline.

Why it matters

For teams running self-hosted AI solutions, selecting the wrong model can lead to silent data corruption. A model that chats well but extracts poorly will create a knowledge base filled with subtle errors that are difficult to detect. By focusing on extraction metrics, engineers can ensure that the foundation of their document AI system is solid before optimizing for conversational flair.

The findings also challenge the assumption that bigger models are always better for local deployment. Smaller models like Gemma 4 12B can outperform larger counterparts in structured tasks, offering significant savings in GPU memory and inference time. This allows small and mid-sized companies to run efficient, accurate document processing pipelines on modest hardware without sacrificing reliability.

Furthermore, the presence of flags such as output cutoffs or runaway repetition provides critical operational insights. Knowing that a model like Llama 3.1 8B frequently hits output limits helps developers set appropriate timeouts and retry logic in their production environments, preventing stalled processes and incomplete data ingestion.

What you can do

• Run the chaoscypher-cli benchmark on your target hardware to verify model performance before deployment, using the --estimate flag to gauge resource usage.
• Prioritize extraction scores over chat scores when selecting a model for document parsing, as this is the primary source of variance in accuracy.
• Check for flags like output cutoffs or ignored settings in benchmark results, as these indicate operational risks that may cause failures in production.
• Consider smaller models like Gemma 4 12B for extraction tasks if they meet accuracy thresholds, reducing infrastructure costs and latency.
• Use the provided confidence intervals to treat top-ranked models as a cluster of viable options rather than seeking a single definitive winner.
• Test models with your own document datasets by adding them to the benchmark tool, ensuring the general results hold true for your specific use case.