Where AI Value Survives: The Untrainable Corner

Summary

A venture investor's framework for understanding where AI value persists as capability becomes cheaper. The piece argues that legible, measurable work gets absorbed — either commoditised by open models or absorbed by frontier labs — while defensible moats cluster in the "untrainable" corner: work where correctness is private, established by judgment inside a customer's data and workflows, and where permission and accountability cannot be transferred to a model. The author illustrates this with coding agents (which lift output 180% but only 30% of shipping), explores why benchmarks measure out the least durable value, and sketches how companies defensibly compete: by embedding inside customer systems, pricing outcomes rather than tasks, and authoring the standards that define what good means in their field. Read the full piece.

Key Claims

• Coding agents increased code output ~180% but only lifted shipping by 30%; the rest of delivery flows through human judgment on system-level correctness that cannot be benchmarked, suggesting the most measurable work commoditises first.
• Measurable correctness gets eaten from both directions: below by task saturation and margin compression, above by frontier labs absorbing the scaffolding (retrieval, routing, tool use) into model weights.
• Private ground truth and permission are not ML problems: a regulator, physician, or firm owner does not grant a model authority via capability; ownership, liability, and trust remain human-side.
• Companies defending untrainable territory avoid thin-wrapper traps by pricing outcomes (resolved tickets, shipped features) rather than tasks, which lets them own the definition of success.
• Authority to define what good means — the benchmark that matters — accrues to whoever is already trusted inside the field (the senior lawyer, the physician, the firm that owns the customer), not to whoever built the smartest model.
• Competition among frontier labs (three-and-a-half players with 5X a development league behind) means no single lab can undercut or absorb every application layer; customers expect choice among suppliers.

Quotes

• "A benchmark is a thing you can measure, and a thing you can measure is a thing you can train against."
• "That kind of correctness can't be read off a leaderboard, and it can't really be read off anything. You find out whether a system that complex works by running it in the world long enough to learn, and a smarter model doesn't make the world run faster."
• "Put it in money terms: a token spent answering a generic question is worth almost nothing, since anyone's model can answer it, while a token spent reasoning over your company's data is worth much more, because it does the thing you actually want, not just the plausible thing."
• "The absorption frontier keeps rising, because we keep learning to measure more of the work, and the measurable gets eaten."
• "The untrainable is value with history. So get inside one, do the unglamorous translation, and start writing down what good means there."

---