Guide topic 03
How Do You Prevent AI From Making Mistakes?
AI can produce incorrect information. A business system should therefore never rely on "the AI said so." Mistakes cannot be eliminated entirely, but they can be caught, limited, and routed to people.
Controls a business system can use
- Approved source material — the AI works from information the company has approved
- Retrieval from authoritative company information rather than general knowledge
- Fixed-rule validation — totals, formats, and required fields are checked by conventional code
- Independent checking — a separate step reviews the result
- Required evidence — answers must point to where the information came from
- Confidence and exception handling — uncertain items go to a person
- Human approval gates before consequential actions
- Permission limits on what the system can see and do
- Business rules the AI cannot override
- Audit logs of what happened and why
- Fail-safe behavior — when something is wrong, stop rather than guess
Match the control to the consequence
Drafting an internal summary may need relatively little control. Sending money, changing pricing, approving contracts, or making commitments to customers should require substantially stronger controls — often including a human approval.
An example from our own work
Western Software Developers' Autonomous AI Knowledge & Publishing System combines AI with validation, independent checking, exception handling, and human review controls where appropriate, so that problems can be caught before anything is published.