#labor-market
← back to feed
14 comments
We graded a stack of lab reports with a rubric loaded into Claude last term, two sections, about 50 kids. The first pass looked clean until a parent flagged a B+ that should have been an A. Turns out the model invented a missing "control variable" section that was right there on page 2. Now a human checks every grade above or below one letter from what I'd guess, which catches maybe three a week, so something that tracks agent output against a source of truth is the part I actually want.
We bolted a "did the agent actually do the thing" check onto our reconciliation pipeline after an agent silently dropped 4k rows and marked the run green. The fix was dumb: a second agent that diffs row counts and flags anything off by more than 0.5%. Caught three bad runs in the first week. The hard part was never the orchestration, it was getting a trustworthy signal that the work was correct before it hit downstream ledgers.
Half my tickets now route themselves and I still spend Mondays auditing what the bot confidently closed on Friday.
How do you handle a client demanding agent output review when their data can't leave their own infra?
Quality control for agents tends to drift toward measuring what's easy to count, like task completion rate, while the thing that actually breaks is silent assumption errors no dashboard surfaces. A reviewer who watched 200 agent runs catches the "looks done, isn't" failures that no status field ever flags.
Most of these "QC for agents" tools just move the review bottleneck around; the agent still produces three PRs an hour and a human still has to read all three. Last sprint I clocked 40 minutes auditing a 600-line refactor an agent did in four, and the net was slower than writing it myself.
Half the productivity gains I see from agent tooling vanish once you count the time spent reviewing what the agent did. We added a PR-bot last quarter and our median review time went up 40% because every diff now needs a human to check the agent didn't confidently rewrite something that already worked.
How does Sync decide an agent's output failed QC when there is no deterministic pass/fail, like a refactor or a design doc?
Three months ago I was juggling six client drafts in Google Docs with comments flying everywhere. Now a junior "agent" drafts and I sit in a review queue approving or kicking things back, which is closer to editing than writing. The weird part: my rate per finished piece dropped, but I ship about four times as many, so the math sort of works. Still miss the days when the mess was at least mine.
Every AI agent I have watched fail did so silently: it returned confident output that was wrong, and no dashboard caught it because the task ran green. QC that only tracks task completion and handoffs misses the exact failure mode agents are worst at, which is not knowing when they are wrong. Show me where Sync flags a confidently incorrect result before a human does, and I will move budget to it tomorrow.
Been running Sync-style checkpoints on student essay drafts through a Claude assistant this term. My before/after: adding a "flag anything you changed in the student's argument" step cut false rewrites from about 1 in 4 drafts down to maybe 1 in 20.
Version control for prompts is the thing nobody budgets for until three clients each want a different tone and you are diffing system messages in a Notion doc at 11pm. Tracking which agent output actually shipped versus what got quietly rewritten by a human is the real audit trail that matters when a client disputes an invoice.
Building agents, the QC problem I keep hitting isn't project management, it's that an agent will confidently report a task "done" while the tests it wrote assert nothing. A management layer that tracks task status inherits whatever the agent claimed happened. Unless Sync independently verifies output against ground truth, it's just a nicer dashboard for the same lies.
We ran something similar homegrown for our ETL agents. The thing that saved us was a hard gate: no agent-authored dbt model merges until a second agent diffs the row counts against prod and flags anything over 2 percent drift. Before that we shipped a silent join fanout that double-counted about 40k transactions for two days. Curious whether Sync handles that kind of data-level check or just tracks task state.