The secret your AI agent leaks before it answers
In August 2026 a group of researchers from MATS, the ELLIS Institute Tübingen, and the Max Planck Institute did something quietly alarming. They collected 6,708 publicly shared AI agent session logs from GitHub and Hugging Face, decoded the "encrypted" reasoning blocks attached to them, and pulled out 62 live API keys, 33 passwords, 24 access tokens, 7 private keys, and 30 personal email addresses. The people who published those logs had scrubbed the visible text first. They still leaked, because the secrets were sitting in a part of the transcript they could not read and did not think to remove.
That paper, Stealing Reasoning Traces from Proprietary LLM APIs (arXiv 2608.09867), was picked up within days by the Cloud Security Alliance and NSFOCUS. It matters to every organisation now wiring large language models and autonomous agents into real workflows, which in the GCC right now is nearly everyone. Banks are running agentic AI in fraud and advisory functions. Telecoms are automating service desks. Government and critical-infrastructure operators are piloting copilots against operational data. The security controls around those deployments are, in most of the shops we walk into, three or four steps behind the enthusiasm.
This is not a "the sky is falling on AI" piece. The cryptography in these systems did exactly what it was designed to do. The failure is subtler and more useful to understand, because it tells you exactly where your own AI risk actually lives. So it is worth slowing down and looking at what really happened.
Three pieces of text, and only one of them is honest
When a modern reasoning model answers a hard question, it produces three separate things. There is the visible answer on your screen. There is a short "thinking" block that some interfaces show while the answer assembles. And there is the full internal reasoning trace, the chain of thought, which can run to thousands of words and is almost never shown to you.
That hidden trace is the interesting one. It holds the model's discarded hypotheses, the raw output of any tools it called, the user data exactly as it was processed, and any secrets that happened to be in the session. Think about what that means for a coding agent asked to strip hardcoded credentials from a repository. To do the job, the agent has to read those credentials. They pass through the reasoning trace before a single word of the answer is written. The visible answer says "removed 4 hardcoded secrets." The trace contains all four in plaintext.
Providers hide the trace for two honest reasons. Commercially, a competitor who harvests enough traces from a strong model has the raw material to train a cheaper imitation, because the trace shows the method, not just the endpoint. On safety grounds, a model sometimes has to reason about a harmful topic in order to refuse it, and that reasoning is filtered out before the visible answer is produced. Neither reason is about protecting your secrets. That distinction is the whole story.
How "encrypted" stopped meaning "private to you"
Here is the design decision that opened the door. A model API is stateless. It remembers nothing between requests, so the continuity you feel in a chat window is an illusion maintained by the client resending the full history every time. The reasoning from turn one still has to be available at turn two. Providers had two ways to handle that. They could store each trace on their own servers and hand the client a meaningless ID, which is simple but means warehousing state for every conversation from every user on the planet. Or they could encrypt the trace, hand it back to the client, and let the client send it back with the next message, storing nothing themselves.
OpenAI, Anthropic, and Google all chose the second path. The encrypted block that comes back is an AEAD envelope, authenticated encryption with associated data, which both hides the content and proves it was not tampered with. On paper that is textbook. The problem is what the envelope authenticates. It proves the block came from the provider and names the model and version. It does not bind the block to the account that created it or the conversation it belonged to. Those fields simply are not in the authenticated data.
When origin is not authenticated, a valid block stays valid everywhere. The researchers documented three flavours of this, each looser than the last: a block replays across sessions, across different user accounts, and across different models in the same family. Cross-model portability is the dangerous one, and it exists for a mundane product reason, so you can switch or auto-route between models mid-conversation.
The extraction, in five unremarkable steps
Anti-distillation training, the thing meant to stop a model disclosing its own reasoning, is expensive, so providers concentrate it on their flagships. The cheaper, faster sibling models in the same family get far less of it. And thanks to cross-model compatibility, the cheap sibling will happily accept a block the flagship produced. That gap is the entire attack.
Read the flow above and notice what did not happen. The strong model was asked one ordinary question. It was never asked to reveal anything, so its refusal training never fired and its output filter saw only a benign answer. The disclosure happens entirely inside the weaker model, which the researchers call a "fuzzy decoder" because it reconstructs an approximation of the reasoning rather than decrypting it exactly. They confirmed the reconstructions were faithful using billing records: the API bills you for the exact number of reasoning tokens consumed, so re-encoding the recovered text and counting tokens gives a number that should match, and across 120 programming problems the counts tracked roughly one to one for Claude.
Four ways this becomes your problem
The paper lays out four consequences. Two of them use blocks the attacker generated, and two use blocks other people published. For an enterprise, the second pair is where the sleep is lost.
Secret leakage. This is the 62-API-keys finding above. Your developers publish agent trajectories for reproducibility, or commit them by accident, and they sanitise the visible text. They cannot sanitise the encrypted blocks, because they cannot read them. Sanitisation only ever touches plaintext, so an "encrypted" block is not scrubbed, it is smuggled out intact. Of the 6,708 sessions scanned, 328 leaked at least one confirmed secret, and that was a deliberately conservative scan.
Prompt injection through resumed sessions. Long agent runs are expensive, so it is tempting to resume a shared one. A block carrying a hidden instruction can be planted in a published trace. In one demonstration, a block instructing the agent to upload files to an external address was ported into an unrelated slide-editing conversation, and the agent dutifully built the slide and exfiltrated the presentation. If your team ever imports third-party agent state, you have imported whatever was hidden in it.
Model and IP theft (distillation). Traces are premium training data for a copycat model, and they are cheap to harvest, roughly $720 to decode 10,000 traces at a budget model's pricing. If your organisation has fine-tuned or heavily prompted a model that embodies real competitive know-how, its exposed reasoning is a leak of that know-how.
Safety-filter bypass (jailbreaking). Models are trained to keep harmful content out of the visible answer, but generally not out of the reasoning, because constraining the trace is thought to hurt safety monitoring. A visible answer can stay perfectly responsible while the recovered trace contains the specific, actionable detail the answer withheld.
What this actually tells you about your AI risk
Strip away the cryptography and three durable lessons remain, and they hold no matter which provider or model you standardise on.
First, "encrypted" is a promise to the provider, not to you. Confidentiality here protects the vendor's IP and integrity stops tampering. Neither guarantees your data is private from a determined third party who gets hold of the block. Treat any reasoning artefact your systems store or transmit as sensitive data in its own right.
Second, the security of a model family is only as good as its weakest member. Hardening the flagship buys you little while a cheaper sibling accepts the same tokens. The same logic applies to your own estate: the least-governed AI integration in your organisation sets your real exposure, not the one the board reviewed.
Third, you cannot clean what you cannot see. Any pipeline that logs, caches, forwards, or republishes model state has to treat opaque blocks as removable secrets, not as safe ciphertext. This is a data-governance problem wearing a cryptography costume.
The table below maps the four attack vectors to the control domains they touch, using frameworks most GCC enterprises are already measured against.
Attack vectorWhat it exposesPrimary control domainFramework anchor
Secret leakage in shared traces
API keys, passwords, tokens, PII
Secrets management, data classification, DLP
ISO 27001 A.8; PCI-DSS Req. 3 & 6; NIST CSF PR.DS
Prompt injection via resumed sessions
Data exfiltration, unauthorised actions
Input validation, agent authorisation, egress control
OWASP LLM01/LLM05; NIST AI RMF Govern/Manage
Distillation / IP theft
Proprietary reasoning and prompts
Access control, IP protection, usage monitoring
ISO 27001 A.5; NIST CSF PR.AC
Safety-filter bypass
Harmful or restricted content
Output governance, red-teaming, monitoring
NIST AI RMF; OWASP LLM Top 10
How we secure enterprise AI in practice
In our engagements we do not start with the model. We start with the data path and the identity, exactly as we would for any other system that touches production. The AI part changes the details, not the discipline. A few of the moves that consistently matter:
Keep real secrets out of the agent's reach. If an agent never needs a production API key in its context, it should never receive one. Scope credentials to the narrowest task, issue them just-in-time, and rotate aggressively. The credentials that leaked in that study were credentials that should not have been in the trace to begin with. A secrets manager, short-lived tokens, and workload identity remove most of the blast radius before any AI-specific control is even considered.
Put a governed gateway between your people and the model. Direct, ungoverned calls from laptops and scripts are where shadow AI grows. A broker or gateway lets you enforce data-loss prevention on the way in, log what left the building, strip or tokenise sensitive fields, and apply per-team policy. It also gives you a single place to reason about which providers and which models are approved, which directly addresses the weakest-member problem.
Apply zero trust to agents and their tools. An autonomous agent calling tools over a protocol such as MCP is a non-human identity making privileged calls, and it deserves the same scrutiny as a human engineer: deny new tools by default, require just-in-time approval for high-risk actions, and log every call with full identity context. Control egress hard, because most agent damage is an outbound request you failed to block.
Decide deliberately where inference runs. For the most sensitive workloads, banking core data, regulated PII, national-infrastructure telemetry, we often land on a self-hosted or private-endpoint deployment, sometimes fully air-gapped, precisely so that reasoning traces and prompts never leave a boundary you control. That is a cost and capability trade-off, not a reflex, and it belongs in the architecture conversation up front.
Treat model state as loggable, sensitive data. Retention rules, encryption at rest under your own keys, and hard controls on who can export session logs. Publishing agent trajectories externally becomes a reviewed action, not a developer convenience.
Monitor, then red-team what you built. Trace monitoring only works while traces stay legible, and the researchers noted that recovered reasoning is often compressed and telegraphic. Assume your monitoring has blind spots and test against them. Adversarial testing of AI systems is now as routine as a penetration test, and should sit on the same cadence.
None of this is theoretical hand-waving. It maps cleanly onto controls our clients are already audited against, ISO 27001, PCI-DSS, SOC 2, the NIST Cybersecurity Framework, CIS Benchmarks, and increasingly the NIST AI Risk Management Framework and ISO/IEC 42001 for AI governance. The industry is converging on the same conclusion. NVIDIA's Open Secure AI Alliance, announced this summer with backing from major vendors, exists to build open tooling for defenders to secure AI, which is a useful signal that "secure the AI itself" has graduated from research curiosity to board-level programme.
A checklist you can run this quarter
- Inventory every place an LLM or agent touches your data, including the shadow-IT ones nobody put on the architecture diagram.
- Confirm no production secret ever enters a model's context; move to scoped, just-in-time, short-lived credentials.
- Route model access through a governed gateway with DLP, tokenisation, and full logging.
- Treat every agent and tool call as a non-human identity under zero-trust: deny by default, approve high-risk actions just-in-time.
- Lock down egress from agent environments and alert on unexpected outbound destinations.
- Classify reasoning traces and session logs as sensitive; encrypt at rest under your keys and restrict export.
- Ban unreviewed external publishing of agent trajectories; strip opaque blocks entirely rather than trusting they are safe.
- For your most sensitive workloads, evaluate self-hosted, private-endpoint, or air-gapped inference.
- Red-team your AI systems for prompt injection and data exfiltration on a fixed schedule.
- Map all of the above to your existing ISO 27001, PCI-DSS, and NIST obligations so AI risk lives inside your real compliance programme, not beside it.
How Aydahwa Enterprise Can Help
We have spent two decades securing infrastructure for banking, telecom, and critical-national-infrastructure clients across the UAE and wider GCC, and AI security is now a standing part of that work rather than a side project. Our team holds the certifications that back it up, including Microsoft Cybersecurity Architect Expert, and we build controls that satisfy the auditors you already answer to.
If you are deploying AI agents and want a clear-eyed view of where your data actually goes, our cybersecurity services cover architecture review, threat modelling, and red-teaming for AI and agentic systems. For decisions about where inference should run and how to isolate sensitive workloads, our cloud security and migration practice designs the private-endpoint and air-gapped patterns discussed above. Day-to-day, our managed IT and security services keep the gateways, secrets management, and monitoring running.
Not sure where you stand yet? Start with our free cyber security self-assessment or work through the cybersecurity readiness checklist, then get in touch and we will walk your specific AI deployments with you. The organisations that get ahead of this are the ones treating their AI systems as production infrastructure today, not the ones waiting for the first incident to make the case for them.
