Human oversight of AI: review and approval that is not box-ticking

“Human in the loop” is the most common answer to AI risk — and one of the easiest to get wrong. A reviewer who approves fifty AI-drafted letters an hour is not providing oversight; they are providing a signature.

The FCA’s good practice publication says firms should decide from risk assessment where human review is needed, and identifies as poor practice review that becomes a box-ticking exercise. This article explains how to design oversight that works.

Where human review is needed

Human review has a cost, and applying it everywhere dilutes attention. Use your use case register and risk tiers to decide:

• Usually no pre-use review: low-risk internal uses such as drafting internal notes or summarising non-confidential material. The person using the AI checks their own output.
• Review before use: customer-facing communications, financial promotions, complaint responses, anything that informs a decision about a customer.
• Approval before deployment: new use cases, new models, and significant changes — through a governance approval with separation of duties.
• Approval before any action: where AI agents can send, submit, pay, trade or update a system of record.

Why oversight fails

Automation bias

When AI output is usually right, people stop checking carefully. Reviewers begin to skim, then to trust. The errors that get through are the ones that look most convincing.

Volume and time pressure

If a reviewer has a queue of hundreds of items and a target time, the review will be fast. Fast reviews are not necessarily bad, but implausibly fast ones are a warning sign.

Not seeing the content

An approval workflow that lets someone approve from a summary, without opening the actual content, invites rubber-stamping.

Unclear standards

If reviewers do not know what they are checking for, they check for nothing in particular.

Self-approval

When the person who requested something can also approve it, there is no independent check.

Designing meaningful oversight

1. Show the content. Reviewers should see the actual draft, the request that produced it and the result of the automated checks — and the system should record that they opened it.
2. Give a checklist. Short, specific checks for the type of item: accuracy against sources, clear and not misleading, appropriate for the customer, promotion rules met.
3. Require a reason. Approvals and rejections should carry a short recorded reason. Rejections should say what needs to change.
4. Separate duties. The person who proposed a use case should not approve it; one person should decide at most one step of a multi-step approval.
5. Use four eyes for the highest risk. Two independent approvals for the most significant decisions.
6. Escalate by risk. High-risk requests should go straight to the right level, not climb a ladder.
7. Set target times — and watch for the opposite. Track overdue approvals, and also approvals made too quickly to have involved real review.
8. Train reviewers. Cover automation bias and what good review looks like. See AI training for staff.

Oversight of AI agents

AI that can act — send emails, update records, initiate payments — needs the strongest controls. A practical rule: AI may draft and prepare, but a person approves any action, and the approval records exactly what was approved. Keep an inventory of agents with their owners, access and controls, as the FCA’s publication suggests.

Measuring oversight quality

Oversight should itself be monitored. Useful measures include:

• The share of approvals where the reviewer opened the content.
• Time taken per review, and outliers at both ends.
• Rejection and amendment rates by reviewer — a reviewer who never rejects anything is worth a conversation.
• Issues found after approval (complaints, corrections) traced back to approved items.

How Dutira helps

Dutira holds high-risk AI requests and drafts for an authorised reviewer, requires the reviewer to open the content and complete a checklist, records the decision and reason, enforces separation of duties and four-eyes approval where configured, and flags very fast approvals as an oversight-quality exception. See Policy & Control.