The eval setup nobody on my team wanted to build

Two-person shop, building a doc-extraction pipeline for invoices. For six weeks we shipped on vibes: I'd tweak a prompt, my cofounder would eyeball ten PDFs, we'd ship. Customer churned in March after we silently regressed on a vendor whose invoices use a two-column layout. The prompt change that helped scanned receipts broke that one customer, and we had no way to know. I spent a weekend wiring up a dumb eval harness. 140 labeled invoices, exact-match on line items, fuzzy on totals, runs in CI on every prompt change. Took maybe 14 hours end to end. Caught three regressions in the first month. The part I want to flag: I resisted this for weeks because it felt like premature infrastructure for a startup. It wasn't. The moment you have more than one customer with different document shapes, your head stops being a sufficient eval. I think founders skip evals because they confuse "move fast" with "don't measure." The teams I see shipping agentic stuff well treat eval sets as the actual product asset. The prompts are disposable.