Evals
Give prompts to coding agents and check what they did.
A prompt can parse fine and still fail when an agent follows it. Evals run your prompts through real coding agents in a sandbox, then check the result.
import {
defineEval,
fileContains,
fileExists,
ranInOrder,
succeeds,
} from "@copybook/cli";
import { setUpWithAgent } from "../src/create-project.prompts";
export const createProjectEval = defineEval(setUpWithAgent, {
agents: ["literal", "claude-code"],
trials: 3,
passRate: 0.67,
sandbox: { template: "evals/template" },
shims: {
neon: { script: "…" }, // record and stub the CLI
npm: "passthrough", // record, then run the real npm
},
assert: [
ranInOrder([/neon init .*--project-id/, "neon deploy"]),
fileExists("neon.ts"),
fileContains(".env.local", "DATABASE_URL="),
succeeds("npm run build"),
],
maxCostUsd: 2,
timeoutMs: 10 * 60_000,
});CI=true npx copybook eval --budget-usd 10How a trial runs
- The
sandbox.templatedirectory is copied into a fresh temp directory. - Shims for the binaries you list go first on
PATH. Every call is logged, whichever agent made it, and a shim can pass through to the real binary or stub it. - The prompt is rendered with
params(and secrets from theirenvvariables) and given to the agent. - After the agent finishes, the assertions run in the sandbox, in order.
A trial passes when the agent didn't time out and every assertion holds. An eval passes when its pass rate meets passRate.
Assertions
| Assertion | Passes when |
|---|---|
ran(pattern) | A shimmed call matches. Calls are matched as bin arg1 arg2. |
notRan(pattern) | No shimmed call matches. |
ranInOrder([a, b, c]) | Calls matching each pattern happened in this order, with others allowed in between. |
fileExists(path) | The file exists in the sandbox. |
fileContains(path, pattern) | The file matches. |
succeeds(command, { exitCode? }) | The shell command exits with exitCode (default 0). |
assertThat(name, check) | Your own check, given the agent run, the calls, and the sandbox path. |
Agents
| Name | Adapter | Notes |
|---|---|---|
literal | literal() | Runs the prompt's embedded commands in order, without an LLM. It's free and deterministic, and if the literal steps fail, no agent will succeed. |
claude-code | claudeCode({ model?, args? }) | claude -p with --permission-mode bypassPermissions, --bare, and --max-budget-usd from maxCostUsd. It reports cost. |
codex | codex({ model?, args? }) | codex exec --json with approvals bypassed. Check the flags against your Codex version. |
Anything that implements AgentAdapter (name, available, version, and run) works too. scriptAgent(name, script) is handy for tests.
Cost and caching
--budget-usdstops starting new trials once the run has spent that much.--history-dir <dir>appends one line per prompt and agent to<dir>/<id>.jsonl, with the pass rate, cost, agent version, and the snapshot hash of the prompt. On the next run, a prompt whose text and agent version haven't changed is reported ascachedand not re-run. Pass--forceto re-run it anyway.--changed --base origin/mainonly evaluates prompts whose rendered text changed.
Transcripts are written to .copybook/results/evals/<id>/.
Safety
Agents get full permissions
Agents run with their permission prompts bypassed, inside a temp directory
that isn't a container. eval refuses to run unless CI=true is set or you
pass --unsafe-local. Use short-lived, scoped credentials in fixtures, never
personal ones.
In CI
The repository's evals.yml workflow shows a full setup:
- It runs nightly and on demand, and on pull requests only when the
run-evalslabel is added. - History is kept on an orphan
copybook-resultsbranch, so git stays the only storage. - Transcripts are uploaded as an artifact.