jcode

Running the standard evals against these harnesses showed us how poorly they represent what agents can actually do. Existing benchmarks face a forced choice. Public benchmarks leak their solutions into training corpora, so scores measure memorization and misrepresent capability. Private benchmarks resist that, but they demand trust in a hidden test set, offer less transparency, and are hard to get access to. Beyond that choice, they share two structural flaws. They impose timeouts, so an agent is penalized for allocating more time to a problem, exactly the long-horizon behavior we want agents to develop. And they score on a coarse, discrete boundary: the gap between two very different models can show up as a single task completion. A model that solves 70% of a bench may sit next to a remaining 30% that demands a far higher capability level, and that entire delta is invisible between the discrete tasks. So we designed a new class of benchmarks with a spec:

• uncontaminatable by construction: there is no answer to memorize, so everything can be public
• hard to saturate by construction: scored on optimization depth, so the ceiling stays open past the frontier
• deterministic by construction: same submission, same score, always
• quantifiable by construction: the metric falls out of the task definition, no rubric, no judge
• continuous by construction: an analog score axis instead of pass/fail, so capability differences show up as score differences at every level, and time is recorded rather than capped
• cheat-resistant by construction: we provide a good reference implementation as the starting point, so web searching for ready-made solutions is only cheating at lower capability levels. At the frontier, there are no implementations left to copy that aren't useful recombinations

This design is immediately trustable because everything is public. Training on a previous model's transcript may even help a little, but it doesn't score better unless the model has genuinely generalized from it: replaying a transcript only matches the old score, and beating it requires optimizing past what was memorized, which is the capability the benchmark measures in the first place.

jcode bench v1 provides three tasks that follow this spec: float-print, json-unescape, and utf16-transcode. We chose these tasks because they grade in seconds and are pure coding on real world functions that are useful and genuinely benefit from optimization: a tight loop the agent can climb, on work that actually ships in libraries. The tasks are also chosen so the implementation cannot be overfit to the test suite: correctness is verified over the entire input space, not a sample. Compare measuring grep's speed by timing it on the Linux repo, where an implementation can be overfit to the Linux repo specifically. Above, we plot the score of the agent over time, from a single user prompt. Note that time is 1:1 correlated with cost and token utilization, because tool call time is negligible in these tasks. Both runs used Opus 4.8 with high thinking, identical prompts, launched concurrently, each stopping on its own judgment. On float-print both passed the full 2³² correctness gate; jcode finished at +8.64 (398x speedup) versus Claude Code's +7.17 (144x).

Jcode's todo tool asks the agent to rate its confidence in each task item, both when the item is assigned and when it's marked done. Tracing those scores across our Terminal-Bench runs showed a clear pattern: confidence is always high after, but sometimes low before, and failures rarely come from tasks the agent was confident about at assignment. The low scores at assignment are real signal, and the jump to 100 at the end is not. So when we see a large spike in confidence, we force the agent to go back and check its work instead of accepting the claim. Ideally, confidence rises incrementally as validation happens during implementation, with each passing test earning a step up. Above, both harnesses on Terminal-Bench 2.1: each line is one trial, from confidence at assignment to confidence at completion. The extra checking pays: trials that finish in time pass more often (92% vs 88%), and even trials cut off by the benchmark's 15 minute limit are more likely to already contain correct work (47% vs 42%).

Agents sometimes stop before the work is done. Jcode checks for unfinished todos and prompts the agent to continue, including in headless `jcode run` sessions. Transient errors are retried, while unrecoverable failures stop the loop.

Jcode is inventing a new form of customizability. One that doesn't limit you to what a plugin or extension can do. Tell your jcode agent to enter self dev mode, and it will start modifying its own source code. Jcode is optimized to iterate on itself. There is significant infrastructure around self development, which allows it to edit, build, and test its own source code, then reload its own binary and continue work in your (potentially many) sessions, fully automatically.