Evals vs. outcomes: why your AI agent should loop on a real-world goal

If you build with AI agents, you have almost certainly been told to write evals. Pick a task, collect a golden set, score the agent against it, and watch the number go up as you iterate on prompts and tools. Evals are good. They catch regressions, they make model upgrades safe, and they give a team a shared definition of "working." But evals quietly answer a narrower question than most people think — and the gap is where a lot of agent projects stall.

An eval asks: did the agent produce the right output on a frozen, pre-labeled task? An outcome asks: did the agent move the real-world number you actually care about? Those are not the same question, and optimizing the first does not guarantee the second.

What an eval can and can't tell you

Evals are measured against a fixed reference. Someone decided, ahead of time, what the correct answer looks like — the right diff, the expected tool calls, the gold summary. That framing is what makes evals repeatable and cheap to run. It is also what makes them blind to three things that decide whether an agent is actually useful:

• Whether the labeled task is the task that matters. A coding agent can pass every unit-test eval and still produce code nobody merges, because "passes tests" was never the goal — "ships a change a reviewer accepts" was.
• Whether the win survives contact with the real world. A reply-drafting agent can score 9/10 on tone and still get zero replies, because the metric that pays rent is response rate, not a rubric score.
• Whether the agent improved or just memorized the eval. Once a benchmark is the target, it stops being a good measurement — the agent (and the human tuning it) overfit to the golden set.

An eval is a proxy chosen by a human before the agent runs. An outcome is the thing the proxy was standing in for.

What an outcome loop is

An outcome loop flips the optimization target. Instead of scoring the agent against a frozen reference, you point it at one real-world metric — organic signups, merged PRs, reply rate, page views, qualified calls — and let it run, ship, and learn from what actually happened. The loop has four moving parts:

• A north star: the real metric that counts as progress, defined once and protected from gaming by explicit guardrails.
• Levers: the mechanisms the agent believes can move the north star, each with its own diagnostics.
• A run: one concrete action on one lever, shipped to the real surface — a repo, a page, an account — not a sandbox.
• Reward: the measured outcome after the action lands, fed back so the next run is a better bet than the last.

The key difference from an eval harness is the feedback source. An eval grades against labels you wrote. A loop grades against reality you observed. Reality is noisier and slower — outcomes compound over days, not milliseconds — but it is the only signal that can't be gamed by overfitting, because nobody got to pre-decide the answer.

Why this is its own discipline

Running an agent once is easy. Running it so each attempt is informed by the last is not — and that gap is what people are starting to call loop engineering. The hard parts aren't the model calls; they're the scaffolding around them: keeping a durable record of what was tried, attaching real reward to it, telling explore (test an uncertain new lever) apart from exploit (rep a proven one), and retiring bets that stop paying off. Do that well and the agent's hit rate climbs run over run. Skip it and you get a very expensive random search.

None of this replaces evals. You still want a fast, offline test suite to catch regressions and keep model upgrades safe — that is exactly what evals are for. The point is that evals are a guardrail, not the goal. Passing them is necessary; it is not the same as winning. The agents that get genuinely better in production are the ones closing a loop on a real outcome, with evals riding alongside as the safety net.

Where to start

Pick one outcome you can actually measure, write down the north star and a guardrail or two, name the levers you think move it, and give your agent a way to read what happened after each run. That is the whole loop. Superdense is the open-source, local-first tool we build for exactly this — it gives your agent durable memory of past runs, reward snapshots tied to real metrics, and the explore/exploit machinery to keep improving. It works with Claude Code, Codex, Cursor, and OpenCode, and everything stays in SQLite on your machine.

If you've been measuring your agents only by whether they pass the test, try measuring whether they move the number. It changes what you build.