A payments firm handles an AI data incident

The firm

Corran Payments is a fictional UK payment institution with around 150 staff, providing payment services to small businesses.

What happened

An operations analyst was reconciling a batch of failed payments. To speed things up, she pasted a spreadsheet extract into a public AI chatbot and asked it to group the failures by cause. The extract included business customers’ names, account details and payment references.

A few minutes later she realised the extract contained customer data and reported it to her manager through the firm’s AI incident route.

How the firm responded

1. Record and triage

The incident was logged with a severity rating, the data involved and the service used. The firm’s incident process sets out who is told and how quickly. Because the report was made promptly, the firm had accurate details from the start.

2. Contain

The analyst still had the chat session open. With the Data Protection Officer’s guidance, she deleted the conversation in the tool and the firm reviewed the provider’s published terms on retention and use of conversations. The firm recorded what it could and could not establish about how the provider would handle the data.

3. Assess the data protection position

The DPO assessed whether a personal data breach had occurred and whether it was likely to result in a risk to individuals’ rights and freedoms, which determines whether the ICO must be notified and whether affected individuals must be told. The assessment, the reasons and the decision were recorded. See the ICO’s guidance on AI and data protection.

4. Consider regulator notification

The firm considered whether the matter needed to be notified to the FCA under SUP 15.3, read with Principle 11. The decision and its rationale were recorded, whatever the outcome. The decision belongs to the firm; the record shows it was considered properly.

5. Find the cause

The investigation asked why the analyst used a public tool. The answer: there was no approved way to use AI for data analysis, the AI policy did not mention spreadsheets, and the team was under time pressure at month end. The person was not the root cause.

6. Fix and close

Remediation actions, each with an owner and due date:

• Make an approved AI route available for operations staff, with a data-classification ceiling and detection of account details and payment references.
• Update the AI policy with explicit examples covering spreadsheets and uploaded files.
• Add a short module on data in files to the AI training, and ask operations staff to complete it.
• Add a statement to the next attestation about using only approved tools for customer data.

The incident was closed only after a second person reviewed the remediation, so no one closed their own incident.

Why reporting worked

The analyst reported the mistake within minutes because the firm had made clear that reporting AI mistakes is expected and supported. Firms within scope of SYSC 18 also need whistleblowing arrangements, and staff can contact the FCA directly. A confidential or anonymous route matters when people are reporting something that involves a colleague.

What the evidence looks like

• The incident record: report, severity, data involved, timeline and decisions.
• The data protection assessment and notification decision, with reasons.
• The regulator notification consideration, with reasons.
• Root cause analysis and remediation actions, owners and completion.
• Second-person closure, recorded in an audit trail that shows the record has not been altered.

What to take from it

• Make reporting safe. The fastest detection was a person who felt able to speak up.
• Fix the cause. The incident showed a missing approved route, not just a careless moment.
• Record decisions, not only actions. Notification decisions need their reasoning on file.
• Close the loop. Policy, training, controls and attestations all changed as a result.

See Monitoring & Risk or read about protecting client data.