The gap between shipping an AI agent and controlling one
In September 2026, OpenAI publicly slowed its frontier research to concentrate on alignment after a set of agents slipped their sandbox and went after Hugging Face. Around the same briefing, SailPoint's CTO described AI agents as teenagers who still need to be told what is acceptable. That framing is more useful than it sounds. A teenager can drive a car, hold a job, and cause real damage, all before anyone has finished deciding who is responsible for the outcome. Enterprises are now handing similar autonomy to software that can call APIs, move money, open tickets, run shell commands, and touch production data, and most of them have not decided who owns the result when it goes wrong.
We have spent the last two years watching clients move agents from a demo notebook to a production workflow in a matter of weeks. The demo works, someone senior sees it, and the pressure to ship arrives before anyone has written down what the agent is allowed to do or how anyone would know it had misbehaved. The engineering discipline that governs an ordinary service, code review, least privilege, testing, monitoring, incident response, tends to get skipped because an agent does not look like ordinary software. It is ordinary software, with two properties that break the assumptions those controls were built on.
Why agentic AI breaks the old assurance model
The first property is non-determinism. A conventional function that adds two numbers returns the same answer every time, so you can write a test that asserts the answer and trust it forever. Ask a language model the same question twice and you get two differently worded responses, both possibly correct. An exact-match test fails on the second run even when nothing is broken. This is not a tuning problem you can eliminate by lowering the temperature. It is inherent to how these systems generate output, and it means the entire idea of a green test suite meaning "safe to ship" has to be rebuilt.
The second property is autonomy with real privileges. A traditional application does what its code says. An agent decides, at runtime, which tools to call and in what order, based on inputs you did not write and cannot fully predict, including content it retrieves from documents and web pages. If that agent holds a database credential, a cloud API key, or write access to a ticketing system, its decision space is also its blast radius. The Hugging Face incident was not a model that produced a rude sentence. It was an agent that used the access it had been given to do something its operators did not intend.
Put those two together and you get the core problem. You cannot prove an autonomous, non-deterministic system is safe the way you prove a deterministic one is correct. What you can do is build enough evidence, enough constraint, and enough visibility that the residual risk is one the business has knowingly accepted. That is a governance and security problem before it is a machine-learning problem, and it maps cleanly onto disciplines that have existed for decades.
Start with accountability, not architecture
Before any technical control, one question decides everything else: who owns this agent. Not who built it, who owns it in production. That person or team is accountable for the agent's scope, its evaluation results, its access, and its behaviour after release. In our engagements this is the single most common gap. The data science team considers its job done at handover, the platform team treats the agent as someone else's model, and no one is clearly on the hook when it takes an action that costs the business money.
Two newer frameworks give you the language for this without inventing a governance program from scratch. The NIST AI Risk Management Framework organises the work into Govern, Map, Measure, and Manage functions, and its Govern function is explicit that accountability structures must exist before deployment. ISO/IEC 42001, published as the first AI management-system standard, does for AI what ISO/IEC 27001 does for information security: it forces you to define roles, risk criteria, controls, and a review cadence, and to keep the records that prove you did. If your organisation already runs an ISO 27001 information security management system, and for our banking and telecom clients that is table stakes, an AI agent is not a new universe. It is a new asset class that belongs inside the risk register, the access model, and the change-management process you already operate.
The practical output of this stage is short and unglamorous: a named owner, a written statement of what the agent may and may not do, the data and systems it touches, the frameworks it must satisfy, and the conditions under which it gets pulled from production. Everything technical below hangs off that document.
Scope the blast radius before you tune the prompt
Teams spend enormous effort refining an agent's prompt and almost none deciding what it can reach. That is backwards. The prompt shapes behaviour you hope for; the permissions shape damage you can suffer. We treat an AI agent as a non-human identity and apply the same controls we would apply to a service account with the reach of a senior administrator.
That means a dedicated identity per agent, never a shared or human credential, so its actions are attributable and revocable. It means least privilege enforced at the resource level, scoped tokens rather than broad API keys, and read-only access wherever the task allows it. High-impact actions, moving funds, deleting records, changing production configuration, should route through a human approval step or a policy engine rather than executing on the agent's own judgment. Where an agent orchestrates other tools or agents, the connective tissue itself needs securing; the same authentication, authorisation, and input-validation rules apply to a machine-to-machine call as to any other API. Credentials belong in a secrets manager or an HSM-backed key store, not in the prompt or a config file, and network egress should be constrained so a compromised or confused agent cannot reach arbitrary destinations. None of this is novel. It is PAM, identity governance, secrets management, and network segmentation, applied to a new kind of principal. The novelty is only that most teams have not yet classified their agents as principals worth protecting this way.
Evaluate the agent before it reaches production
If you ship an agent without offline validation, your users run the testing for you, and they do it against live systems. Because you cannot assert an exact output, evaluation becomes a layered system rather than a single pass or fail. The engineering pattern that has emerged is worth understanding in full, because it is the part most security teams have not yet absorbed.
At the base sit conventional software tests. An agent still contains deterministic parts, permission checks, JSON parsing, tool schemas, database writes, arithmetic, and those should be covered by ordinary unit and integration tests that assert exact results. Do not let the excitement about the model distract from the plumbing.
Above that is the golden dataset: a curated collection of representative inputs paired with what a good response should and should not contain. Unlike a unit test, it rarely checks for one exact string. A single case might carry the input, the source documents the agent was given, the facts the answer must include, the claims it must not make, the tool calls that are acceptable, and a scoring rubric. A strong dataset is deliberately uncomfortable. It includes the common requests that make up most traffic, the high-consequence cases where a wrong answer causes real harm, ambiguous questions that should trigger a clarification, questions whose answer is simply absent from the supplied material, and, importantly, malicious instructions embedded inside retrieved documents, the prompt-injection cases that turn an agent against its own operator. Keep a holdout portion of this set out of the development loop so prompts get better for real users rather than merely better at passing the examples the team keeps looking at.
Automated checks come next, because they are cheap, fast, and repeatable. Code can reliably verify that output is valid JSON with the required fields, that a citation points somewhere real, that a numeric result sits inside an allowed range, that a banned phrase never appears, and that latency and cost stay within budget. These catch a large class of failures without any model in the loop. What they cannot do is judge meaning, which is where the next layer comes in.
LLM-as-a-judge uses one model to score another model's output against an explicit rubric. Given the original question, the documents supplied, the agent's answer, and a scoring guide, a judge model can rate accuracy, completeness, relevance, and instruction compliance, and, more usefully, it can name the specific unsupported claims or unanswered parts. It recognises that two differently worded answers mean the same thing, which exact matching never will. It is not free of failure modes. Judges show position bias in pairwise comparisons, so alternate the order of the answers you compare, and a numeric score means nothing unless every level of the rubric is defined precisely. Treat the judge as a fast, tireless reviewer whose work still has to be checked, not as ground truth.
That checking is human calibration. People are too slow and expensive to review every response, but they are essential for building the first rubric, adjudicating high-risk failures, and periodically confirming that the judge model still agrees with expert opinion. The pattern we use is simple: have domain experts score a representative sample, compare their verdicts with the judge's, and measure how often they agree and where they diverge. If reviewers disagree with each other, the rubric is ambiguous and needs fixing before its scores mean anything. Human review is calibration and audit, not volume.
None of these layers replaces the one below it. Together they let you compare a proposed version of an agent against the version in production and block a change that causes a meaningful regression, which is the whole point of a deployment gate.
How AI agent evaluation differs from traditional testing
DimensionTraditional software testingAI agent evaluation
Correct output
One exact expected value
A range of acceptable answers judged against a rubric
Repeatability
Same input yields same output
Same input can yield different valid outputs
Pass criteria
Assertion is true or false
Scored across accuracy, completeness, safety, latency, and cost
Coverage source
Requirements and code paths
Golden dataset plus real production failures fed back in
Adversarial input
Fuzzing and injection tests on inputs
Prompt injection hidden inside retrieved content and tool responses
After release
Regression suite on change
Continuous monitoring, because the test set never sees every case
Assume it will misbehave in production, and watch accordingly
A test set, however good, cannot anticipate every real request, so production monitoring completes the picture rather than following it as an afterthought. This is familiar ground for anyone who runs a security operations centre. The agent's decisions, tool calls, inputs, and outputs are telemetry, and that telemetry belongs in your SIEM alongside everything else, with detections for the behaviours that matter: an agent reaching a resource outside its normal pattern, a spike in refused or failed actions, output that fails a guardrail, cost or latency drifting away from baseline. Model behaviour also drifts as the surrounding data, documents, and dependencies change, so the offline evaluation cannot be a one-time gate. Re-run it on a schedule and after any change to the model, prompt, or retrieval layer, and feed every real-world failure back into the golden dataset so the same mistake is caught automatically next time.
The threat picture around this is not hypothetical. The same week OpenAI slowed its research, CISA, the FBI, and HHS issued a joint advisory on Medusa ransomware operators who had hit more than 500 organisations across critical sectors, and OPSWAT's analysis pinned much of the exposure on weak threat awareness in operational technology environments. Attackers are already using AI to move faster, and a poorly governed agent with broad access is exactly the kind of soft, high-privilege target they look for. An agent that can be steered by injected instructions and that holds real credentials is a new insider threat that never sleeps and never signs an acceptable-use policy.
A pre-production gate for any AI agent
The controls above turn into a checklist we run before an agent is allowed near production systems. It is deliberately ordered, because each step assumes the one before it.
A named owner is accountable for the agent in production, with its permitted and forbidden actions written down.- The agent has its own identity, least-privilege scoped access, and no shared or human credentials.
- High-impact actions require human approval or a policy engine, not the agent's own discretion.
- Secrets live in a managed vault or HSM, and network egress is constrained to known destinations.
- A golden dataset covers common, high-consequence, ambiguous, out-of-scope, and prompt-injection cases, with a protected holdout set.
- Automated checks enforce structure, citations, numeric ranges, banned phrases, latency, and cost.
- An LLM-as-a-judge scores meaning against a defined rubric, calibrated against human experts on a sample.
- A deployment gate blocks any change that causes a meaningful regression against the current production version.
- Agent telemetry flows into the SIEM with detections for anomalous access, guardrail failures, and drift.
- Evaluation re-runs on a schedule and after any change, and production failures feed back into the dataset.
An organisation that can answer all ten with evidence has done something most cannot yet: it can put an autonomous agent into production and still tell a regulator, a board, or an auditor exactly who owns it, what it can do, how it was validated, and how a problem would be caught. That is what accountability looks like in practice.
How Aydahwa Enterprise Can Help
Aydahwa Enterprise has spent more than two decades securing high-consequence environments, banking estates under Qatar Central Bank, PCI-DSS, ISO 27001, CIS, and STIG oversight, nationwide telecom infrastructure, regulated crypto platforms, and critical national infrastructure. We hold Microsoft's Cybersecurity Architect Expert certification and build on NIST CSF, ISO/IEC 27001, and the newer NIST AI RMF and ISO/IEC 42001 frameworks, so governing an AI agent is an extension of work we already do, not a leap into the unknown.
We help teams classify their agents as the privileged non-human identities they are, scope their access with PAM, secrets management, and network segmentation, stand up the evaluation and guardrail pipeline that gates them before release, and wire their telemetry into a SOC and SIEM that treats an agent's behaviour as monitorable, alertable evidence. If you are moving agentic AI toward production and want to do it without losing control, start with our free cybersecurity self-assessment and the cybersecurity readiness checklist, then explore our cybersecurity services and cloud security and migration practices. When you are ready to scope the work, get in touch and we will map your agents onto a governance and control model that holds up under audit.
