Evaluations and Benchmarks
Evaluation frameworks, test harnesses, empirical bakeoffs, and comparative benchmarks for scoring and validating LLM Wiki and agent memory architectures.
Agentic Memory Index (Verging Labs)verginglabs.comAn independent benchmark and comparative evaluation suite testing 12 agent memory architectures (including Karpathy Wiki, GBrain, Cognee, and Mem0) across 56 working sessions. Evaluates direct recall, cross-conversation synthesis, temporal reasoning, conflicting facts, and erasure, measuring the accuracy, answer speed, and token cost trade-offs of compiled markdown vaults against vector memory.

An empirical benchmark evaluating OpenWiki as a persistent memory layer across a 111-document synthetic corpus. Demonstrates that a compiled wiki yields a ~5x accuracy gain over search alone on multi-source reasoning, while materializing citations into a typed evidence graph increases accuracy by a further 27%, doubles citation consistency, and cuts cost per correct answer by 58%.
Evaluating OpenWiki with WikiBench (LangChain Blog)langchain.comAn evaluation overview introducing WikiBench, a benchmark running inside the Harbor framework to measure wiki usefulness for agents. Demonstrates that pairing a generated wiki with codebase access yields higher task correctness and lower token costs than querying raw source code alone.
memory-arenapypi.orgAn open-source benchmarking suite and evaluation harness designed to test long-term agent memory strategies across interdependent, multi-session tasks. It establishes a standardized empirical testing framework comparing classical vector storage, graph-based abstractions, and flat-file architectures like the Karpathy LLM Wiki pattern—measuring recall accuracy, context degradation, and token overhead.
The LLM Wiki Got a Spec Before It Got a Benchmark (Alden DoRosario)adorosario.github.ioAn empirical benchmark evaluating 14 LLMs generating Open Knowledge Format (OKF) markdown documents across a four-stage compilation loop. Uses a cross-vendor judge and per-call cost ledgers to analyze price-performance plateaus, token efficiency, and generator failure modes like empty-body artefacts and metadata schema crashes.
