A literature graph found relevant papers—but could not reliably tell whether they closed an open problem

The useful system and the failed judge

A new two-layer knowledge graph shows that literature retrieval and research-status judgment are separate benchmark problems. The system preserved curated catalogs of 1,053 database-research problems and 1,000 AI problems, then attached a rebuilt bibliographic evidence layer. It linked at least one evidence work to 579 database problems and 592 AI problems, including 770 works outside the database catalog across 336 problems. But the paper’s audit found that only 9 of 41 model-produced solves or refutes verdicts were fully correct—21.95%—while 30 were partly correct and two were wrong.[1]

That distinction matters because the retrieval layer did produce measurable gains. On the catalog-only benchmark, the rebuilt graph preserved all 47 answerable database-query results exactly. For AI, 34 of 35 queries returned the same rows; the remaining difference came from a tie under LIMIT 10, and removing the limit produced the same 11 papers. The evidence layer also grounded 1,265 of 5,058 extracted database-paper problems to catalog entries, a 25.0% grounding share; the AI run grounded 1,311 of 6,885, or 19.0%.[1]

“Refutes” did not mean the problem was closed

The central audit checked every high-stakes verdict rather than a sample. The status checker ran over 572 database problems and 1,653 papers, plus 578 AI problems with at most eight papers per problem. Across 3,856 graded evidence links, it emitted seven solves verdicts and 34 refutes verdicts.[1]

None of the 34 refutes actually disproved an open problem. Eight of 32 AI refutes correctly contradicted a claim on the relevant problem page, but that is not equivalent to closing the problem negatively. Only one of seven solves closed the central question in the problem statement; the other six settled restricted sub-cases.[1]

The most frequent error was logical rather than merely factual: nine of 41 verdicts treated a paper showing that current methods fail as a refutation of an open “is it possible?” question. Another six verdicts settled only a sub-case, and four refuted a rival explanation or intermediate result rather than the open problem itself.[1]

The guardrail is part of the result

The graph did not silently convert those judgments into catalog facts. Its architecture records status changes only as proposals with evidence, leaving the curated layer untouched. Four suggestions were written, but no status changed.[1]

The authors also limit what the audit establishes. One author judged the 41 outputs with model assistance using titles, abstracts, quoted evidence and problem statements rather than full papers; the audit measures precision on emitted verdicts, not recall over all genuine solutions the system may have missed.[1]

The practical lesson is sharper than “LLMs make mistakes.” A system can retrieve relevant literature, preserve an existing query benchmark and still fail at the final entailment step that decides whether a research question is closed. For living research catalogs, status mutation should remain a review queue—not an automated field update.

Source

[1] Mandarapu and Kunkunuru, “Rebuilding a Field’s Open-Problem Catalog from the Literature: A Two-Layer Knowledge Graph, and Why Status Must Stay Proposed,” arXiv v1, Oct. 29, 2026.