LLMs can faithfully reproduce just 27.3 % of encoded research concepts, according to the newest lineage‑reasoning benchmark. The figure comes from a systematic reconstruction task that treats each paper as a set of typed “Idea Genome” objects and asks models to recover the exact inheritance pattern. This stark drop from fluency on surface text signals a compositional bottleneck that has been invisible in conventional evaluations.
Before IdeaGene, most assessments measured token‑level accuracy or free‑form summarisation without probing whether systems understood how ideas evolve across papers. Benchmarks such as MMLU or HELM treat each question in isolation, ignoring the genealogical structure that scientists actually exploit when extending prior work. Consequently, progress on those metrics has been taken as evidence of deeper scientific reasoning, even though the underlying ability to recombine learned concepts remained untested.
The strongest system reaches only 27.3 % exact accuracy on lineage reasoning. The authors report that “The strongest system reaches only 27.3% exact accuracy on lineage reasoning, and structured lineage context reshuffles system rankings rather than helping every participant uniformly” [1]. This result is measured on IG‑Exam, a suite of 42 task types where an exact match of the Idea Genome reconstruction is required.
Even the best direct LLM, GPT‑5.5, attains merely 23.1 % exact accuracy, while augmenting it with Claude Code lifts performance to the benchmark ceiling of 27.3 % (Table 4) [1]. The modest gain from a tool harness highlights that simply adding external code does not close the compositional gap; the underlying model still fails to capture most inheritance relations.
Structured lineage context reshuffles rankings but does not uniformly lift systems. The study notes that “structured lineage context separates systems rather than lifting them uniformly,” indicating that providing richer genealogical cues helps some models on specific sub‑tasks (e.g., Inheritance Tracing) while leaving others (e.g., Lineage Verification) essentially unchanged [1]. This asymmetry suggests that current architectures are sensitive to prompt framing but lack a robust mechanism for integrating multi‑step idea evolution.
Even the best system reaches only 27.3 % exact accuracy on IG‑Exam; tool scaffolds help T2 Inheritance Tracing but barely move T4 Lineage Verification; and structured lineage context separates systems rather than lifting them uniformly [1]. The authors therefore conclude that a compositional bottleneck limits scientific synthesis, and they leave open whether richer pre‑training on genealogical data or novel prompting strategies can raise the ceiling.
Practitioners building AI‑assisted literature review pipelines should adopt IdeaGene‑Bench as a sanity check for any model claimed to synthesize research ideas. Until a system consistently exceeds the 27 % reconstruction threshold, downstream applications risk propagating fragmented or incoherent scientific narratives. Running IG‑Exam on new fine‑tuned checkpoints will surface whether architectural tweaks genuinely improve compositional reasoning rather than merely inflating surface fluency.
