RTGL found future users inside three RelBench task tables

A benchmark can have a clean train/test split and still leak the future through the SQL that builds its labels. A new paper introducing the Relational Task Generation Language (RTGL) reports exactly that failure in three RelBench tasks: driver-dnf, user-engagement and user-post-comment.

The authors rebuilt tasks from RelBench’s rel-f1 and rel-stack datasets as declarative RTGL queries, then joined the generated tables against RelBench’s reference tables. In driver-dnf, they say the original SQL checked whether a driver was active without imposing an upper bound at the prediction timestamp. That let later activity affect earlier examples and could include drivers who did not yet exist. The two Stack Exchange tasks similarly failed to filter users by creation date, producing examples for users who had not registered at the prediction time. The paper does not quantify how many rows were affected, so this is evidence of a task-definition flaw—not yet a measurement of its effect on published model rankings. Source: RTGL paper

The artifact clears the paper gate

This result is unusually inspectable. The RTGL repository contains the Python compiler, temporal and static validators, tests, three experiment notebooks and an MIT license; it is also packaged for pip install rtgl. A companion task repository is linked from the paper. RTGL compiles a PREDICT … FOR EACH … specification into SQL and can either return that SQL or execute it with DuckDB. Temporal aggregations are defined relative to explicit prediction timestamps rather than hidden inside bespoke task-building code.

The authors also trained GraphSAGE and HGT models on generated task tables. On the corrected rel-f1 driver-dnf test split, GraphSAGE reported 0.7219 ± 0.0123 accuracy and HGT 0.6932 ± 0.0138. Those values establish that the generated table can support downstream training; they should not be read as an improvement over scores produced from the old labels because the evaluated examples have changed. Source: RTGL paper, Table 2

What benchmark maintainers should copy

The important contribution is not another model score. It is moving temporal boundaries into a machine-checkable task language and shipping the generated SQL for inspection. RelBench maintainers can now diff corrected task tables, count affected rows by split, and rerun models before replacing any leaderboard values.

RTGL is not a universal replacement yet. The authors say deeply nested multi-hop tasks without direct foreign-key paths still need intermediate views, and they identify large-dataset performance as future work. But for the class of errors it found, the remedy is concrete: version the task specification, generated SQL and resulting table together. A benchmark score is only as clean as the query that decided which rows existed at prediction time.