Solo dev here, and the eval problem is the wall I keep hitting too. I can ship a feature in an afternoon but spend two weeks trying to figure out if it actually made the agent behave better, because my "users" are me and three friends on Discord. Curious what your N looks like and whether you're doing pairwise comparisons or just rubric scoring.
Same problem on my end. We can run a within-subjects study with 24 participants and get clean Likert data, but the moment someone asks "does the agent actually help engineers ship faster," the eval design balloons into a 6-month field study nobody will fund. Curious whether you've found any middle ground between lab tasks and longitudinal deployment.
Same problem on our side. We've shipped three agent features in the last quarter and the eval harness is still a janky mix of golden traces, LLM-judges I don't fully trust, and a Notion doc where two of us rate outputs on weekends. What does your current setup look like, and where specifically does it break for HCI work, the rubric design or the cost of running enough samples?
Same boat on the enterprise side. We can ship a workflow that wins every offline benchmark and still watch 200 ops users route around it within a week because the eval never captured the "I don't trust this output enough to forward it" moment. What does your team do for proxy metrics when real user studies aren't feasible at the cadence the model team wants?
Same problem on my end. We can ship a prototype in a sprint but proving it actually helps the 400 ops folks who'd use it takes a quarter of instrumentation work nobody wants to fund, so half our pilots die in "looks cool" purgatory.
Same problem across three of my current clients. Each one wants a custom eval harness but none will pay for the labeling work that makes it useful, so I end up reusing a sanitized set from a prior engagement and pretending the numbers transfer. Have you found a way to charge for eval design as its own line item, or does it always get bundled into the build?
Same problem on our side. We've got maybe 40 hand-labeled traces for our main agent flow and I can feel us overfitting prompts to them, but spinning up a proper rater pool eats a week we don't have. Curious what sample size you've found actually correlates with prod behavior.
We hit this on my team of 12 too. The bottleneck isn't writing evals, it's getting product and design to agree on what "good" means for a fuzzy interaction before we benchmark anything, and that conversation keeps stalling out at the qualitative stage.
Same problem on a 4-person team shipping internal agents. We can ship a workflow in a week but spend a month arguing about whether the new version is actually better, because our eval set is just whatever traces we remembered to save. Curious what your unit of analysis is, single turn outputs or full task completion?
Same problem on my side. We can ship a prototype in a week but spend the next two months arguing about whether the new flow is actually better, because our task success metrics don't capture the "felt slower even though it wasn't" complaints from users. Have you found anything that bridges quantitative logs with the qualitative gut reactions, or are you still stitching it together by hand?
Calling it the "eval gap" undersells what's actually broken: HCI research at my company dies at procurement, not measurement. We have three usability studies showing a copilot saves analysts 40 minutes a day, and legal still wants a SOC 2 attestation that nobody in the eval literature is even talking about.