Why it matters
Tests are organizational memory of incidents and policy. The March refund bug should still fail in December. Suites onboard engineers faster than oral history.
Flaky full-text equality tests get muted. Stable assertions on sources, citations, and prohibited claims endure.
Muted flaky tests are worse than no tests. Invest in stable assertions on sources and citations instead of brittle full-string equality on prose.
Every production incident should add a test before closure or the incident was wasted.
How it works
Tag tests by risk tier and owner. Critical tests block all releases.
Run suites on PR, nightly, and pre-release with production-like permissions. Staging with overly open indexes lies about safety.
Add tests from every production incident and major support escalation theme.
Integrate failures into CI with clear on-call ownership.
Run weekly staging tests even without releases to catch environment drift.
Run tests with production-like permissions in staging weekly even without releases. Open staging indexes lie about safety.
Tag tests with owners in CI notifications. Unowned suites decay silently.
Example
Nintendo runs fifty refund test prompts before each Help Center deploy. A failed assertion on pro-rata language blocks release until the Refund Policy mirror updates.
Common mistakes
- 1Tests that only check substring match without citation rules
- 2No tests added after production incidents
- 3Staging tests skipped for urgent hotfixes
- 4Flaky tests teams mute instead of fix
Test the answer before your customer does.
Run answer tests and evaluations against your governed knowledge before you ship.
See answer evaluations