jcode bench

Optimize production-grade primitives. Exhaustive correctness is required. Every verified improvement counts, and runs have no time limit.

How a run works

Each task gives the agent a working implementation, an exhaustive verifier, and a published deterministic cost model. The agent edits the code and submits it for grading as often as needed. Each grade takes seconds, and the benchmark records every new best result. Correctness on every possible input is the gate. Among passing submissions, the score measures doublings of cost reduction relative to the starting implementation.

Tasks

Three tasks, chosen for fast grading and substantial optimization headroom. A better result raises the known frontier for every later run.

Frontier model comparison →

Leaderboards: every harness and model, by task

Every run below is one solo agent that passed the official final correctness grade. Reasoning effort is shown where a campaign includes multiple levels. Each chart plots the running best score against active agent time. The two default lines are the original 2026-07-05/06 jcode and Claude Code head-to-head runs. The buttons add or remove any other harness, model, or reasoning effort.

These charts rank task by task across harnesses. For the model-only ranking with a single combined score, see the model comparison: six frontier models through the same pinned Jcode harness, aggregated by geometric-mean doublings.

json-unescape

float-print

utf16-transcode

Best is the highest grade sampled during the run, so it can sit slightly above the official final score, and for float-print not every sample is a full 2³² gate. Runs were recorded on different dates with different harness versions, and each cell is one run. †The Claude Sonnet 5 json-unescape run logged a single +14.30 grade from a degenerate random corpus; the surrounding grades sat near +0.93 and no later grade came close, so it is excluded here and documented in the raw curve data.

All task specs, graders, verifiers, and given implementations are public at github.com/1jehuang/jcode-bench. There is no hidden test set. Read about the benchmark class at /bench.