SRL benchmark finds reuse costs 2.15× to 6.44×—but recursion dominates both engines

A new benchmark of two Shape Rules Language engines reports a consistent speed advantage for a dedicated evaluator—but its most useful finding is where that advantage shrinks. Across 25 synthetic workloads, Eyeleng 1.2.2 ran 2.15× to 6.44× faster than a Comunica-based SRL engine, @comunica/query-shacl-rule 1.0.0. The paper, posted September 3, says Eyeleng was faster on every workload and found no crossover point.

The comparison is reproducible

The authors timed both JavaScript engines on the same AMD Ryzen 5 5500U machine with 15 GiB RAM, Linux 6.8.0 and Node.js 20.20.2. Timing began with the rule set already in memory and ended after full closure was materialized, so both measurements included parsing, dependency analysis, stratification, fixpoint evaluation and result materialization—but excluded process startup, file reads and serialization.

Most workloads ran 25 times after one discarded warm-up. The six largest received between five and 22 timed runs. Results are medians with interquartile ranges, and both engines produced identical output-triple counts on every workload. The public performance directory exposes the harness, generated rules and W3C test runner.

Reuse has a measurable fixed cost

The largest reported gap came in the firing-count suite. With 3,000 rules over a fixed workload of 6,000 input and 6,000 output triples, Comunica’s median was 967.4 ms against 150.2 ms for Eyeleng: a 6.44× speedup. Fitted per-rule costs were 193 microseconds for Comunica and 20 microseconds for Eyeleng. The authors attribute that difference to sending every rule body through the full SPARQL pipeline—construct parsing, algebra construction and planning—rather than evaluating it directly.

That penalty was smaller when data dominated. At 100,000 input triples with two rules and one fixpoint pass, Comunica took 7,862 ms versus Eyeleng’s 2,390 ms, a 3.29× gap.

Recursion changes the verdict

On a 25-node transitive-closure chain, the gap narrowed to 2.15×: 65.6 ms for Comunica and 30.5 ms for Eyeleng. At 200 nodes, both slowed sharply—to 34,510 ms and 14,135 ms—because both repeatedly evaluated all rules over the accumulated store. The paper measures cubic growth and argues that semi-naive evaluation would likely matter more here than whether the engine reused SPARQL infrastructure.

Conformance also complicates a speed-only reading. Eyeleng passed all 166 tests; Comunica passed 154, deriving the correct triples in every positive test but accepting 12 inputs that conforming implementations should reject. This is therefore not a universal RDF-engine ranking. It is a controlled implementation trade-off: SPARQL reuse buys formats and query infrastructure quickly, while a dedicated evaluator removes substantial per-rule overhead. For recursive workloads, however, changing the fixpoint algorithm is the more important optimization target.