Tests as data

Why it matters

A Thoughtworks author, after interviewing the consultancy's developers earlier this year, reports a striking loss of confidence in automated test suites: developers said AI coding agents refactor both production code and tests as features evolve, so tests reviewed at creation no longer guarantee correct behavior later. The piece argues that when AI writes most code, developers shift from being testers to being customers of that code, which makes customer tests—the human-AI contract—far more important than developer-facing unit tests. AI-generated tests, it notes, tend to be voluminous, hard to read, tightly coupled to implementation through heavy mocking, and prone to merely confirming existing behavior, including bugs. The author's proposed remedy is to move the contract into test data rather than test code, using human-readable formats: acceptance tests drawn from Extreme Programming, test DSLs, tabular testing in the style used by the Go community, and approval testing's textual state representations. Since AI can now cheaply build parsers and runners that were once costly to maintain, readable data-driven tests become practical.