Deleting a vector record may not delete its influence on retrieval

A vector database can stop returning a deleted record and still produce a measurably different evidence set because that record once existed. That is the central result of Ghost Echoes, a black-box audit of deletion in retrieval-backed applications. The distinction matters for any analytics agent that retrieves documentation, metric definitions or examples before generating SQL: deleting the source object is not necessarily the same as restoring the retrieval behavior of a system in which that object was never inserted.

What the experiment measured

The paper’s primary experiment used ChromaDB 0.4.24, all-MiniLM-L6-v2 embeddings and a synthetic corpus of 500 privacy-sensitive person records. It selected 18 targets across dense, medium and sparse semantic clusters and repeated each condition over three seeds, producing 54 paired observations. Retrieval depth was fixed at Top-5.

The authors first checked ordinary deletion correctness across five backends. Across 270 trials—18 targets, five systems and three seeds—the deleted identifier did not reappear in the Top-40 results. That verifies the visible API contract tested by the paper; it does not establish that every storage-layer trace vanished.

The harder test compared the Top-5 evidence before and after deletion. On ChromaDB, target deletion produced a median retrieval-centroid drift of 0.1522, versus 0.0412 when the system deleted a non-target record from the same semantic cluster. Target drift exceeded that control in 53 of 54 paired comparisons, with a Wilcoxon p < 0.001. Vocabulary shift showed the same ordering: 0.285 after target deletion versus 0.095 for the same-cluster control.

The control is the story

The study cannot directly observe the true counterfactual—an otherwise identical index in which the target was never inserted. It approximates that state by deleting a nearby, non-target record that was outside the original Top-5. That is a stronger control than comparing deletion with doing nothing, but it remains a proxy. The result should therefore be read as evidence of target-specific retrieval drift under the tested setup, not proof that all vector databases retain deleted meaning.

The attack result is similarly bounded. With five target-relevant queries, deletion detection reached 61.1% in the evaluated setting. The attacker is assumed to know or strongly suspect a deletion and to probe the target’s semantic neighborhood; the experiment does not reconstruct the deleted text.

Why “rebuild the index” is not yet an answer

In the paper’s local mitigation test, rebuilding the index without the deleted records left median drift at 0.1522, indistinguishable from its reported baseline. A light embedding perturbation reduced drift only to 0.1451 at epsilon no greater than 0.02; stronger perturbation reduced it to 0.1089 but sharply damaged NDCG@5. Those are useful negative results, but they cover one evaluated rebuild design rather than every vendor’s maintenance path.

For retrieval-backed SQL agents, the operational lesson is concrete: a deletion acceptance test should check both identifier absence and the stability of nearby retrieval contexts. Until that second check passes, “deleted” describes the API result—not necessarily the agent’s evidence behavior.