Skip to content
hendrik kühnel
DaiVELOPER

run 5, nine recorded runs

What it measured

Twelve questions, answered by three arms, counted in approximate tokens.

The harness does no sampling, no model calls and no randomised ordering, so all token counts, byte counts, call counts and derivability flags are byte-identical across run 4's four recorded runs and run 5's five — nine in all, including one after rm -rf node_modules dist && npm ci, and five taken across run 5's work with the later ones after each fix landed. The spread on the score is exactly zero by construction. That rules out flakiness in the harness. It rules out nothing about the world.

Read RESULTS.mdwhat it is not

the headline

3,486 against 16,495, at 12 of 12 derivable

Against a scripted agent using native tools, a 78.9% reduction. Against the cheapest possible native sequence — an oracle handed the answer in advance, which is a floor rather than a competitor — 64.0%. Charge the 2,000-token session preamble the native arms never pay and it is 5,486, still under both.

Derivability is not the arm's own assertion. Each scenario declares ground-truth anchors as a file plus a substring, resolved to a line number against the committed fixture at run time; an arm scores only if every anchor's line reached context with its file and line number attached, and the treatment's claimed ranges are re-read from the fixture and checked.

Approximate tokens ingested across twelve benchmark scenarios, one bar per arm. A scripted native-tool baseline costs 16,495; the cheapest possible native sequence, given the answer in advance, costs 9,677; DaiVELOPER costs 3,486, or 5,486 once the once-per-session tool preamble is charged. All four arms answered 12 of 12, checked by re-reading the fixture rather than taken from a tool's own claim. The preamble row is the same treatment with the 2,000 tokens of tool descriptions a session pays once, which the native arms never pay.
Approximate tokens ingested per scenario, RESULTS.md section 1. Counts are o200k_base, a proxy for Claude's tokenizer rather than the thing itself. Median per-scenario margin against the baseline is −64.1%; median absolute saving 405 tokens.
scenariobaselineoracleDaiVELOPERvs baseline
def-format-currency1,56752132−91.6%
decoy-disambiguation1,567109132−91.6%
last-declaration-of-long-file1,0689691−91.5%
exports-of-long-file7,7896,851957−87.7%
module-exports-ledger44844897−78.3%
declined-charge-behaviour680120221−67.5%
call-chain-checkout1,5921,000628−60.6%
interface-shape-settlement-report1918476−60.2%
return-type-reconcile-settlement540392341−36.9%
error-to-status-mapping532303382−28.2%
callers-of-post-ledger-entry291103234−19.6%
cross-module-usage-audit230119195−15.2%
total16,4959,6773,486−78.9%

the composition

The aggregate hides its own composition

95.2% of the margin over the native floor comes from one scenario of twelve. Remove exports-of-long-file and the remaining eleven total 2,529 against the floor's 2,826 — 10.5% under it before the preamble, and above it once the preamble is charged. Against the baseline the concentration is milder and still real: one scenario is 52.5% of the win and three are 74.6% of it.

So: decisively cheaper than an agent grepping and reading files. Whether it is cheaper than the best possible native sequence rests, on this benchmark, on a single question — and a twelve-scenario benchmark with one dominant term is a thin base for a general claim.

How the 6,191-token margin over the cheapest native sequence divides. One scenario, exports-of-long-file, accounts for 95.2% of it; the remaining eleven scenarios together account for 4.8%. Removing that one scenario leaves the eleven at 2,529 approximate tokens against the native floor's 2,826 — under it, and above it once the session preamble is charged. The question that scenario asks is 'list every export of this long file with its return type', and no native tool can do it at all. That is a real capability difference and it is also one question out of twelve.

six of twelve

Where it loses to the floor

Against the baseline, nowhere: all twelve scenarios are negative, from −15.2% to −91.6%. Against the oracle, six are positive. The pattern is legible — it wins where structure cannot be recovered from text, and loses where the answer is one line and the questioner already knows its exact shape.

def-format-currency, +153.8%
The oracle greps ^export function formatCurrency\( and gets one line. find_symbol returns three declarations, marks the near-miss and states what it scanned. It costs 80 more tokens to rule the decoys out than an oracle spends knowing in advance there is nothing to rule out.
callers-of-post-ledger-entry, +127.2%
Grep is genuinely good at a well-chosen call-site pattern. What find_callers adds is the enclosing function per site and an auditable count of what it excluded. Unchanged since run 2 and probably close to irreducible.
declined-charge-behaviour, +84.2%
The oracle does not search at all: it reads eight lines at an offset it was told. Nothing that has to find the branch can compete with that, which is precisely why the oracle is a floor and not a competitor.
cross-module-usage-audit, +63.9%
The same shape as the callers scenario, against a narrower glob. Structure costs what structure costs.
error-to-status-mapping, +26.1%
The one remaining two-call sequence: one call to locate, one to read. A combined locate-and-read would remove the first response. Named as the most likely remaining win in run 3 and not done.
decoy-disambiguation, +21.1%
The same call as def-format-currency. The oracle's grep has to be wider here, so the gap nearly closes.

Not '78.9% cheaper'. 78.9% cheaper than an agent that has to search, and structurally capable of things no grep can do, at a small premium over a grep that already knows the answer. That second condition is the oracle's defining privilege, and no real agent has it.

what the machine pays

The tokens are the model's cost, not the whole cost

The treatment opens more files than the baseline and spends four times the wall clock, because prefilter-then-parse has to read every candidate file in order to rule it out. That is the trade the project makes on purpose: milliseconds of local CPU for thousands of tokens of model context. At 118 ms for twelve scenarios it is invisible; on a repository two orders of magnitude larger it might not be.

Per-benchmark totals across run 4's four runs, RESULTS.md section 3. The benchmark's treatment arm is deliberately index-free — a fresh workspace per call, re-reading from disk exactly as the native arm does — so the token comparison is not flattered by a cache the native arm has no equivalent of.
armfile opensbytes off diskwall clock
baseline552872,53028.5–29.7 ms
oracle197373,4979.3–9.8 ms
DaiVELOPER588830,331116.1–118.3 ms
A fresh server process per sample, timed from initialize completing to the first outline response, median of five. Run 5 re-measured the fixture on Node v24.16.0 on macOS, different hardware from run 4's container, so both rows are here and neither replaces the other. End-to-end numbers are smaller than the index's own because a fresh process pays transport setup and a stat sweep over every file whether the cache hits or not. The cache lives outside the served root and deleting it is always safe.
repositorycoldwarmspeedup
bench/fixture, 45 files · run 522.7 ms4.4 ms—
bench/fixture, 45 files · run 426.9 ms4.20 ms6.4×
unrelated repo, 120 files · run 497.7 ms6.64 ms14.7×

Run 5 made the warm index slower and says so. Storing each file's import and re-export bindings — the thing that makes call attribution possible — grew the cache from 56 KiB to 75 KiB, and the warm path is dominated by reading that file: a 7.5–10.5× warm speedup across run 5's five runs, against run 3's 17.7–19.1×. Cold build 8.7–9.6 ms, warm 0.9–1.2 ms, on 45 files.

End to end through a fresh server process, where grammar compilation dominates, that regression is invisible — which is the table above. Both are on this page because quoting whichever one flatters is the thing this site is trying not to do.

threats to validity

Written trying to break the result

the scenarios chose themselves
Twelve questions, written by the people building the thing that answers them, against a fixture written by the same people. No held-out set, and no third party wrote a single question. RESULTS.md calls this the largest threat and does not resolve it. The one declared control — module-exports-ledger, where no win was expected — came in at −78.3%.
the fixture was built to be hard to grep
formatCurrency on 64 lines across 21 files, a 605-line file with its answer last, a four-hop call chain and two decoys, all asserted by the generator. The properties are documented up front rather than discovered afterwards, and the token numbers have still only ever been measured on this one repository.
a proxy tokenizer stands in for the real one
Every count is o200k_base, which is not Claude's tokenizer. Ratios measured the same way are meaningful; absolute values are not. The byte ratio, −78.0%, tracks the token ratio closely enough that the conclusion does not hinge on it — which does not eliminate the concern.
no model was in the loop
The metric is whether the evidence reached context, not whether a model reading that context answers correctly. A structured response could in principle be harder to read than raw file text and this benchmark would not notice. The live A/B has never run: the CLI in that container returns Not logged in.

In order: scenarios written by someone else against a repository nobody here chose; a live A/B with a real model and real usage counts; the benchmark run on a second repository; an adversarially-tuned baseline, which PROJECT.md explicitly invites and nobody has taken up.

Until at least the first two exist, the defensible claim is narrow: on twelve questions of its own choosing over a fixture of its own construction, DaiVELOPER delivered the same evidence as a scripted native-tool agent for 21% of the approximate tokens, and did so reproducibly.