CyberCQR Case study · AI security and governance
Automation without blind trust: testing and guarding AI before it runs the joiner, mover, leaver process
How a customer services organisation with high staff turnover, which wanted more automation, starting with how people join, move and leave, put testing, guard rails and human approval around both generative and agentic AI before letting it act.
About this case study. Neil Brady consulted on and led this work for a customer services organisation with high staff turnover and large seasonal recruitment of temporary workers. The organisation is not named.
Summary
The point for boards. An AI agent that can create accounts or change payment details has the same power as the person whose job it replaces, but it can be misled in ways a person would notice. Automation is safe when the organisation can show that the AI behaves consistently, cannot act alone on anything that matters, and is checked regularly for tampering.
The value
| Value | What it meant here |
|---|---|
| Time and cost | The aim was to cut onboarding from more than a week to a few days. For seasonal workers employed for 10 to 12 weeks, that is a large share of their time back as productive work, every season. |
| Risk not overlooked | The saving is only real if the agent cannot create shadow users, change bank details or leak data it should not. Human approval, testing and contamination checks meant the automation could go ahead without trading the saving for a new risk. |
| Repeatable | The testing, guard rails and checks became standing procedures, so the next process to be automated starts from a proven approach rather than from scratch. |
Table 1. The value of the work, in risk, cost and simplicity.
The starting point
The organisation runs customer services, with high staff turnover and large seasonal recruitment of temporary workers. Onboarding and training were the areas where it most wanted efficiency, so it wanted more automation, starting with the joiner, mover, leaver (JML) process: setting people up when they arrive, changing their access when they move role, and removing it when they leave.
Its main platforms for staff were Workday, the system of record for people and HR data including pay details, and Salesforce, where customer service staff did their work. Both held user accounts that the JML process had to create, change and remove.
Getting one person ready to be productive meant dozens of steps across dozens of platforms, documents and processes: creating and managing accounts, assigning training, adding people to groups and giving them the right role-based access (RBAC) roles. It took more than a week. With seasonal workers employed for only 10 to 12 weeks, that was a large share of their time with the organisation. The aim was to bring it down to a few days, with an AI agent building and carrying out all the onboarding and offboarding steps.
At that volume JML is attractive to automate, because it is repetitive and rule-based and happens every week. It is also one of the most sensitive processes in any business, because it decides who has access to what. With many temporary staff arriving and leaving each season, an account created in error, or one that is never removed, is easy to miss.
Two kinds of AI were in scope. Generative AI answers questions and drafts content from the organisation's information. Agentic AI goes further: it takes actions in other systems, such as creating an account, assigning training or adding someone to a group. The onboarding agent needed both.
The risks Neil highlighted
Before any automation went near live systems, Neil set out what could go wrong if the AI were given the rights the process needs. Each risk matches an entry in the OWASP Top 10 for large language model applications [2].
| Risk | What it could look like | OWASP 2025 |
|---|---|---|
| An agent acting beyond what was intended | Creating shadow user accounts that nobody asked for, or adding or changing bank account details in the HR and pay system, from a misleading or manipulated instruction | LLM06 Excessive agency [3] |
| Poisoning and contamination | Content slipped into the model's training or reference material that changes how it answers or acts | LLM04 Data and model poisoning |
| Sensitive content reaching the wrong people | Confidential information added to the AI's sources and then served to users who do not have permission to see it | LLM02 Sensitive information disclosure |
| Manipulated inputs | Instructions hidden in a document, email or ticket that the AI reads and follows | LLM01 Prompt injection |
| Unchecked outputs | An AI response passed straight into another system without being validated | LLM05 Improper output handling |
Table 2. The risks, and the OWASP Top 10 for LLM Applications 2025 entries they correspond to.
An agent that can create a user can create a shadow user.
What was put in place
The controls were based on the OWASP guidance for large language model applications [2], applied to the organisation's own processes.
| Control | What it does |
|---|---|
| Human approval before any change | Agents prepare actions but cannot commit them. A person reviews and approves each change before it is made, which is the control OWASP recommends for high-impact actions [3]. |
| A control set of questions and procedures | A fixed set of test questions and procedures is run regularly, and the results are compared with the expected answers, to confirm the AI still gives consistent results. |
| Testing for model drift | Test plans that detect when the AI's behaviour changes over time, for example after a model update or a change to its data. |
| Input and output sanitisation | Inputs are checked and cleaned before the AI acts on them, and outputs are checked before they are shown to users or passed to other systems. |
| Guard rails, constantly reviewed | The limits on what the AI may say and do are reviewed on a schedule and after any change, not set once and forgotten. |
| Contamination and poisoning checks | Protocols and procedures to check the model and its sources regularly for poisoning, contamination and content that should not be there. |
Table 3. The controls.
Showing, not telling
The most persuasive part of the work was a demonstration. Neil showed how quickly a language model could be poisoned, or have sensitive content added to it, and then serve that content to users who did not have permission to see it. Seeing it happen made the case for the human approval step and the regular contamination checks far better than any written risk assessment.
Published research points the same way. In 2025, Anthropic, the UK AI Security Institute and the Alan Turing Institute found that about 250 malicious documents were enough to plant a hidden behaviour in models of every size they tested [1].
The outcome
The organisation gained a set of protocols and procedures to run continually: testing for drift and consistency, sanitising what goes in and out, reviewing the guard rails, checking for contamination, and keeping a person in the loop for every change an agent proposes. Automation could go ahead with the risks understood and controlled, rather than discovered later.
What we would add now
- The OWASP Top 10 for Agentic Applications. Published in December 2025, it covers risks specific to agents, such as goal hijacking, tool misuse and identity and privilege abuse [4].
- The UK AI Cyber Security Code of Practice. The government's code sets baseline security principles across the AI lifecycle, from design to end of life [5].
- Agents as identities. We would give each agent its own identity with the least access it needs, so its actions are logged and its rights can be reviewed like any other account.
Lessons for a firm of 500 to 2,000 people
- Start with what the AI is allowed to do, not with what it could do. Give it the least access the task needs.
- Where turnover is high, watch the leavers. Automation that adds people quickly must remove them just as reliably.
- Keep a person in the loop for anything that creates accounts, changes access or touches payment details.
- Build a control set of test questions before go-live, and rerun it after every model or data change.
- Treat the AI's sources as sensitive. Anything added to them can be repeated to anyone who can ask.
- Show the board the risk. A short live demonstration is more persuasive than a long risk register entry.
Glossary
| Term | Meaning |
|---|---|
| Agentic AI | AI that takes actions in other systems, such as creating accounts or updating records, not only answering questions. |
| Generative AI | AI that produces text, answers or other content from the information it has been given. |
| Guard rails | Rules and checks that limit what an AI system may say or do. |
| Human in the loop | A design where a person must review and approve an AI's proposed action before it takes effect. |
| Joiner, mover, leaver (JML) | The process of giving people access when they join, changing it when they move role, and removing it when they leave. |
| Model drift | A change in how an AI system behaves over time, for example after an update or a change to its data. |
| Poisoning | Deliberately adding content to an AI's training or reference data to change how it behaves. |
| Role-based access control (RBAC) | Giving people access through roles that match their job, rather than one permission at a time. |
| Shadow user | An account created outside the normal, approved process, often without anyone noticing. |
Table 4. Terms used on this page (alphabetical). Who these organisations are: see OWASP and Anthropic on our references page.
Further reading
From CyberCQR
- Could your AI be turned on you? Our board summary on the risks of AI agents that can act in your systems.
- AI Data Leakage Through Context Exposure. How AI assistants can reveal information to the wrong people.
- AI security and governance. How we work with firms on this today.
References
- [1]Anthropic, with the UK AI Security Institute and the Alan Turing Institute, A small number of samples can poison LLMs of any size, 9 October 2025.
https://www.anthropic.com/research/small-samples-poison - [2]OWASP Gen AI Security Project, OWASP Top 10 for LLM Applications 2025.
https://genai.owasp.org/llm-top-10/ - [3]OWASP Gen AI Security Project, LLM06:2025 Excessive Agency.
https://genai.owasp.org/llmrisk/llm062025-excessive-agency/ - [4]OWASP Gen AI Security Project, Top 10 risks and mitigations for agentic AI security, December 2025.
https://genai.owasp.org/2025/12/09/owasp-genai-security-project-releases-top-10-risks-and-mitigations-for-agentic-ai-security/ - [5]Department for Science, Innovation and Technology, AI Cyber Security Code of Practice.
https://www.gov.uk/government/publications/ai-cyber-security-code-of-practice
Security is the outcome, not a solution.
CyberCQR is a UK cyber security advisory practice. We help organisations adopt AI with the testing, guard rails and governance to trust it, without vendor commissions or product resale.