>ceryce_

// evaluation methodology

Evaluation

For months I've been building an agentic assistant for my own use, and the harder half of that work turned out to be evaluation — figuring out whether a change actually made the thing better, rather than just feeling like it did. This page is that half, on its own.

// why this, not vibes

An agent will tell you it's fine. That's not evidence.

Every agentic system I've built has an opinion about how well it's working, and that opinion is cheap to produce and easy to believe — it's confident, it's articulate, and it's occasionally wrong in ways that are expensive to catch late. So the actual engineering problem, underneath whatever the system is nominally for, is building measurement that doesn't trust the system to grade itself: a baseline before a change, a prediction registered before a result, a judgment that comes from somewhere blind to what answer it's "supposed" to give.

None of what follows is a research paper. It's the methodology I built and used because I needed to actually know — not guess, not vibe-check, know — whether a design decision held up.

// the throughline

Six rules I hold myself to

  • Baselines before optimizing

    I measure what a system does with nothing added before I credit whatever I add next. Skip the baseline and every improvement is a story you tell yourself.

  • Shadow mode before any behavior changes

    A new decision-maker runs alongside the old one first — classifying, logging, changing nothing — until its accuracy is evidence and not a hope. Flipping it on is a separate, later decision.

  • Blind A/Bs over self-graded ones

    If the thing being evaluated can also grade its own homework, the grade is worthless. The judgment has to come from somewhere that doesn't already know the answer it wants.

  • Pre-register the kill condition

    I write down what result would prove me wrong before I collect a single data point. A prediction with no way to fail is not a prediction — it is a hope wearing a lab coat.

  • Watch for metric traps

    A per-unit efficiency number can get worse while the whole system gets better — more checking, more overhead, more looking things up — and the number that matters is the outcome, not the proxy that's cheapest to watch.

  • Measurement that holds up

    A finding that only survives inside the document that produced it isn't a finding. I try to leave a re-runnable check behind, not just a paragraph that sounded right at the time.

// the work

Six evaluations, applied

Hallucination measurement54 artifacts · 0 clean

Measuring confidently wrong claims

An agent I run stated assumptions in the exact same confident register as things it had actually verified — same voice, wildly different provenance. Instead of writing another rule telling it to be careful, I built a measurement layer first: tag every reply with what was actually consulted, then audit every fix ever proposed for this failure across the project's history. Fifty-four written countermeasures, seventeen with a documented later recurrence, zero with an evidenced clean record. The fix that survived wasn't a better sentence — it was a disclosure derived mechanically from what a turn actually did, because a self-reported "I checked X" is just another unverified claim wearing a badge.

  • Measurement design
  • Instrumentation
  • Root-cause analysis
Long-horizon agent evaluation3 arms · 1 simulation

Constitution vs. recall, scored head to head

Does an externally-authored identity — a tracked charter with rails and its own eval suite — hold an agent together over a long horizon better than giving it a better memory? I reimplemented a published long-horizon business simulation from its own paper (the vendor never released a runnable harness), then ran three arms on the identical sim and seed: a bare baseline, the baseline plus a real memory system, and the baseline running under a versioned charter re-read from disk every turn. The baseline came first on purpose — you can't credit a constitution with anything if you never measured what the model does without one.

  • Benchmark reimplementation
  • A/B/C design
  • Long-horizon coherence
Behavioral persistence study~21-session campaign

Pre-registering a prediction, then scoring it

Does a rule survive across a long conversation because it was stated once, or because a reminder of it stays near the model at every turn? I wrote the exact numbers that would confirm the hypothesis and the exact numbers that would kill it, before running a single session — then ran a campaign of roughly twenty-one sessions to test it for real. Both pre-registered falsifiers were checked directly, and neither one fired. Writing the failure condition down first is the only thing that makes a later "we confirmed it" worth trusting.

  • Pre-registration
  • Causal design
  • Statistical scoring
Memory evaluationBlind A/B · human oracle

What's safe to forget, tested against a human, blind

Any system that decides what to keep and what to let go needs a way to find out if it decided right. On a sampled recall, the agent answers twice — once with a retrieved memory, once without — and I pick the better one blind, without being told which is which, and say why. That verdict is the ground truth the cheap signals (access counts, sentiment proxies) get calibrated against, never the other way around.

  • Ablation testing
  • Human-in-the-loop
  • Calibration
Deployment methodologyLog-only · zero branching

Shadow mode before any behavior changes

Before a lightweight classifier gets to decide anything on its own, it runs in shadow: it scores every message, writes its verdict to a log, and the full system keeps running exactly as it did before — no branch, no shortcut taken on its say-so. Turning it from an observer into a decision-maker is a separate, later call made on the accuracy evidence the shadow run produced, not on how good the design looked on paper.

  • Progressive rollout
  • Local inference
  • Risk gating
Agent evaluation gates7/7 admission gates

Every specialist earns admission the same way

Each purpose-built subagent clears the same fixed battery of regression evals before it's trusted with anything live — a full pass, not a spot check, and not a person's judgment call overriding a failed gate. New capability doesn't get a pass because it seems fine; it clears exactly what the last one had to clear.

  • Regression testing
  • Admission gates
  • Agent governance

// so what

The point was never the assistant

Every one of these started as a question about one system, and every one of them generalizes: how do you know a change to an agent's behavior actually worked, rather than just changed? Pre-register the prediction. Hold something back as a baseline. Get a blind verdict when you can. Ship in shadow before you ship for real. And when a metric starts looking better, check whether the thing it's supposed to be standing in for actually got better too — because it often hasn't.

More on the assistant these evaluations were built around is on my portfolio.