A 41-million-row test says database explainers can agree on fidelity—and still disagree on the columns

A new benchmark of column-level explanations for relational deep learning finds that the explanation method can change which database attributes look important—even when standard fidelity scores look similar. The result matters for teams using model explanations to audit leakage, justify predictions or decide which schema fields deserve scrutiny.

The primary paper compares four global attribution methods: single-column ablation, Monte Carlo Shapley, Kernel SHAP and Mask Learning. The authors test each method on eight entity-prediction tasks drawn from four RelBench databases, using both heterogeneous GraphSAGE and a Relational Graph Transformer. Every database contributes one binary-classification task and one regression task.

The scale is real, but the scope is narrow

The four datasets range from the 98,000-row, 67-attribute Formula 1 database to the 41.3-million-row, 128-attribute event database. The clinical-trial database contains about 5.5 million rows and 140 attributes; the arXiv database contains about 2 million rows and 21 attributes. The study therefore covers materially different schema sizes, but only graph-based relational models and eight tasks.

Across those tasks, regression explanations generally agreed more strongly than classification explanations. Single-column ablation and both Shapley approximations often produced similar rankings, particularly among the highest-ranked attributes. Mask Learning diverged more often because it optimizes for a jointly sufficient subset rather than estimating each column’s marginal contribution.

That distinction is not cosmetic. The paper reports a clinical-trial regression case under RelGT where Kernel SHAP and Mask Learning had a ranking-similarity score of 0.68 at the top-20% cutoff, yet their fidelity results were tightly clustered. In other words, two explainers could identify different columns while looking similarly faithful under a predictive-degradation test.

The simple method was much cheaper

Runtime widened the practical gap. At each method’s empirically observed convergence point, Kernel SHAP ran a median 37× faster than Monte Carlo Shapley. Single-column ablation then ran a median 9.3× faster than Kernel SHAP. The authors explicitly leave a more comprehensive runtime study for future work, so those ratios should be read as results for this setup—not universal speed claims.

The benchmark’s negative finding is the useful one: fidelity alone does not certify that two explanation methods tell the same story about a database. Fidelity measures whether a selected subset preserves or disrupts predictions; it does not require competing methods to select the same subset.

For practitioners, the paper supports a two-part audit. First, compare the actual top-column rankings, especially for classification tasks. Second, use fidelity as a complementary test rather than a substitute for that comparison. Where ablation and Shapley agree, the 9.3× median runtime result makes simple ablation an attractive baseline. Where they disagree, the system should expose that instability instead of presenting one attribution list as ground truth.

The limits are equally important: the study does not report a human evaluation, and the experiments do not cover relational foundation models or tabular transformers. The benchmark establishes that explainer choice can matter; it does not establish which explanation users should trust.