we cover >the future of work_

about
back to feed

The eval gap nobody on my team wants to own

I run a small HCI research group, four researchers plus two engineers. Six months ago we started using agentic coding tools for our analysis pipelines. Productivity on first drafts roughly tripled. Time-to-publishable-result barely moved. The bottleneck shifted from writing code to trusting it. Our pipelines do things like cluster qualitative codes or compute inter-rater agreement on annotated transcripts. The agent produces something plausible in twenty minutes. Verifying it took one of my engineers most of a week, because we had no eval harness for the messy intermediate steps, only for the final numbers. Nobody wants to own that eval work. It's unglamorous, doesn't generate publications, and the engineers feel it's the researchers' methodological responsibility, while researchers feel it's an engineering concern. So we keep shipping faster drafts that take just as long to defend. My current hypothesis is that whoever in your org used to do code review is now doing eval design, whether they signed up for it or not. If that role is vacant on your team, your productivity gains are imaginary. You've just moved the debt.
30·aisha_kapoor·

1 comments

0meeraIyer·1mo
Calling evals "nobody's job" is the cope. On our team it landed with data eng by default because we're the only ones who can wire prompt outputs to the warehouse and join them against labeled ground truth from Snowflake, and pretending otherwise just means six more weeks of vibes-based regression checks. Stop waiting for an Eval Engineer req to open and put it in the data eng JD.