A 91% data-cleaning score falls to 18% when the film names become fictional

The result

A new controlled benchmark shows how much an LLM data cleaner can depend on recognizing the entities it is asked to repair. In CoRe, DeepSeek-V4-Pro repaired 91% of erroneous movie-year cells when the rows contained real films, but only 18% when the same rows used fictitious English-word names; syllable-based names reduced accuracy to 13%. Crucially, the year candidates remained available after renaming, so this drop cannot be explained solely by removing an external identifier mapping. The result appears in Figure 1 and the evaluation discussion of the CoRe paper.

What CoRe changes

The authors build 21 controlled tables: nine variants each for Hospital and Movies across three construction seeds, plus three for Beers. Original, word-surrogate and syllable-surrogate conditions preserve rows, error positions and non-identity values while replacing names and identifiers. A separate fresh-record condition applies the same error mechanisms to records outside the benchmark. The study evaluates four commercial and four open-weight LLMs, alongside Baran and TabClean, using about 23,000 prompts (paper, Sections 3–5).

The effect extends beyond one year column. At construction seed 0, the four commercial models’ identity-column repair F1 on Movies was 0.17 to 0.42 higher for fresh real films than for word-based surrogates. On Hospital, three commercial models lost 0.59 to 0.72 repair F1 on identity columns after syllable replacement. A screened continuation probe found no positive evidence that the tested models had memorized complete benchmark rows, which narrows the interpretation: familiar entity knowledge, rather than demonstrated row recall, can still lift a cleaning score (paper, Sections 6–8).

The benchmark can reward the wrong thing

CoRe also tests whether models preserve assigned values when those values conflict with familiar facts. Even after explicit instructions to use table evidence, commercial models rewrote assigned directors of known films toward real-world credits at rates from 0.000 to 0.404. Under the standard prompt, that range rose to 68%–100% (paper, Section 9).

Reference data introduce another failure mode. On Rayyan, an ISSN correction matching a title-linked PubMed record received no credit, while 434 of 461 credited TabClean repairs—94%, excluding development rows—reproduced the benchmark’s reference date transformation (paper, Section 10).

The practical lesson is not that public cleaning benchmarks are useless. It is that one aggregate repair score can mix table reasoning, familiar-entity knowledge and agreement with imperfect references. CoRe’s paired renaming conditions make those components visible; future cleaning claims should report them separately.