The eval setup nobody runs is the one that would actually matter

I run a small HCI lab, four PhDs and two RAs. For the past year we've been studying how product teams adopt coding agents. The pattern that keeps surprising me: every team has eval harnesses for output correctness, almost none have evals for the human side. Time-to-trust, abandonment after a bad suggestion, the silent reroute where a senior just stops using the tool on Fridays because it burned them Monday. We instrumented one partner team of 11 engineers for six weeks. Agent acceptance rate looked healthy at 38 percent. But 60 percent of rejections came from three people, and those three were the ones doing architectural review. Nobody noticed because the dashboard aggregated. The productivity story the leadership saw was a lie of averages. My take: until eval suites treat the operator as a variable, not a constant, the numbers we publish about agent performance are closer to marketing than measurement. I don't see anyone shipping this. It's tedious, the IRB is annoying, and it makes your tool look worse before it looks better.

▲ 28·jiwoo_lee·2mo

5 comments

0aminataDiallo·2mo

The evals I want are multi-turn, multi-day runs against a real staging env with realistic tool failures injected, and nobody on my team of six has time to build that harness. So we keep shipping on single-turn rubric scores and finding the actual regressions in production three days later.

0CamilaTorres·1mo

The evals we skip are the ones that need real users, real data, and a week of cleanup before they're runnable. My team built a nightly suite a year ago and it still only covers the cases we already knew how to write fixtures for, which is roughly the easy half.

0MateoSilva·1mo

Genuine question from someone two years in: how do you even start building the "real" eval set when you don't yet know which failure modes matter? Every time I try, I end up with toy cases that pass and the actual prod weirdness shows up a week later in Slack.

0diego.rivas·1mo

Everyone on my team writes evals against the happy path because that's what you can score automatically. The stuff that bites us in review (subtle context misuse, plausible but wrong refactors) needs a senior to grade it, and nobody's funding that.

0aminataDiallo·1mo

The "nobody runs it" framing lets everyone off the hook. I run a 30-student weekly rubric on my district's tutoring bot, and the reason colleagues skip it isn't difficulty, it's that admin already bought the license and a bad eval becomes a political problem. The eval that matters is the one your procurement team can't ignore.