Open-weight models are now good enough to run in-house. That changes the security question.
For most of the last three years, the practical choice for enterprise AI was a hosted API from a handful of vendors. That is no longer the only option. Open-weight model families such as NVIDIA's Nemotron, Meta's Llama, Mistral, and Qwen have closed enough of the gap that a mid-size reasoning model running on your own GPUs can handle document analysis, code assistance, and retrieval-augmented search at a quality most regulated teams would have paid an API bill for last year. NVIDIA alone now publishes some of the most downloaded models on Hugging Face, and it ships the training data and recipes alongside the weights, not just the finished file.
That shift is welcome for a bank, a telecom operator, or a utility that cannot send customer data to a third-party endpoint in another jurisdiction. It also moves a large amount of risk back onto your side of the fence. When you host the model, you own the model supply chain, the inference infrastructure, the data that flows through it, and every control that used to be the vendor's problem. We've spent the last several engagements helping clients stand these deployments up, and the pattern is consistent: teams get a model answering questions in an afternoon, then spend the next two months discovering everything they skipped. This article lays out what to secure before you go live, mapped to the standards auditors will actually ask about.
Why regulated organizations are bringing models in-house
The motivations are rarely about chasing a benchmark. They are about control and jurisdiction.
Data residency is the first driver. Under the UAE's Personal Data Protection Law (Federal Decree-Law No. 45 of 2021), GDPR in Europe, and sector rules like banking confidentiality regimes, sending regulated data to an inference endpoint you do not control creates a transfer and processing question that is hard to answer cleanly. Running the model inside your own boundary removes the cross-border transfer from the equation entirely. Second is confidentiality of the prompts themselves. In our banking work, the prompt is often more sensitive than the answer, because it contains account context, transaction detail, or internal case notes. Third is availability and cost predictability: a self-hosted model has no per-token meter and no dependency on a provider's rate limits during an incident. Fourth, and increasingly, is contractual. Government and critical-infrastructure clients are writing "no external model calls" straight into their requirements.
None of those benefits are free. You trade a vendor's security team for your own, and the threat model is genuinely different from the web applications your SOC already knows how to defend.
The attack surface you inherit
A self-hosted model introduces failure modes that traditional application security testing does not cover. The useful reference points here are the OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS, the adversarial threat knowledge base for AI systems. Between them they name the categories that matter for an on-prem deployment: supply chain compromise, data and model poisoning, prompt injection, sensitive information disclosure, excessive agency, and unbounded resource consumption.
Read those against your existing controls and the gaps show up quickly. Your web application firewall does not understand prompt injection. Your data loss prevention tooling was not written to inspect a model's free-text output. Your change management process has no concept of a model weight file as a deployable artifact with its own provenance. Each of those needs a deliberate answer.
Trusting the model supply chain
A model downloaded from a public hub is executable content, not a static document. This is the risk OWASP labels LLM03 (Supply Chain) and LLM04 (Data and Model Poisoning), and it is the one teams underrate most.
Two concrete problems come up in practice. The first is serialization format. Older model checkpoints distributed as Python pickle files can execute arbitrary code the moment they are loaded, which turns "we downloaded a model" into "we ran an unknown binary on a GPU host with network access." Insist on the safetensors format, which stores weights without executable payloads, and reject pickle-based checkpoints unless they have been scanned and converted in a controlled environment. The second is provenance. A model named to look like a well-known release, uploaded to a public repository by an account you have never verified, is a classic typosquat. Pin the exact repository, the exact revision hash, and mirror the artifact into an internal registry so production never pulls live from the public internet.
Treat the model like any other third-party dependency. That means a software bill of materials for the model and its tokenizer, cryptographic hash verification on every pull, malware and pickle scanning in the pipeline, and a documented approval step before a new model version reaches production. If you already run this discipline for container images under your ISO 27001 or SOC 2 program, extend the same pipeline rather than inventing a parallel one.
Where the model runs, and how it is isolated
The inference host is a high-value target. It holds the model, it sees every prompt, and it usually has access to whatever data source powers retrieval. Design the placement accordingly.
For the most sensitive workloads we default to an air-gapped or tightly egress-controlled zone. The GPU hosts sit in their own network segment with no outbound internet path, so a compromised model or a malicious prompt cannot exfiltrate data to an external server. Where a full air gap is impractical, an explicit egress allowlist and monitored proxy achieve most of the same protection. The inference server itself, whether you run vLLM, NVIDIA NIM, TensorRT-LLM, or Ollama, should be hardened to a known baseline. We apply the relevant CIS Benchmarks to the host OS and container runtime, drop the container's Linux capabilities to the minimum, run it as a non-root user, and keep the model files on read-only mounts.
Access to the inference endpoint deserves the same rigor as any internal API. Authenticate every caller, authorize per application rather than issuing one shared key, put the endpoint behind your existing identity provider, and rate-limit aggressively. That last control maps to OWASP's LLM10 (Unbounded Consumption): without request and token ceilings, a single misbehaving client or a denial-of-wallet style attack can saturate expensive GPU capacity and take the service down for everyone.
Governing the data that flows through the model
The model is only part of the system. Most enterprise deployments wire the model to internal data through retrieval-augmented generation, and that plumbing is where confidentiality breaks down if nobody is watching.
Three questions decide whether your deployment is defensible. Who is allowed to retrieve what? A retrieval layer that ignores the caller's permissions will happily surface a document the user was never cleared to see, which is a data breach dressed up as a helpful answer. Enforce the source system's access controls at retrieval time, not after. What gets logged, and for how long? Prompt and response logging is essential for a SOC and for incident investigation, but those logs now contain the same sensitive data as the prompts, so they inherit the same classification, encryption, and retention rules. We route them into the SIEM with the appropriate sensitivity label rather than leaving them in a flat file on the inference host. And where does regulated data physically live? Vector databases and embedding stores are data stores like any other, subject to the same residency and retention obligations as the source records they were built from.
Handled well, this is also where a self-hosted model earns its keep under a privacy regime. Because nothing leaves your boundary, the data-transfer analysis that complicates a hosted API largely disappears, and your ISO/IEC 27701 privacy controls apply cleanly.
Runtime threats: prompt injection, output handling, and agents
Prompt injection (LLM01) remains the hardest problem in the category, and there is no single control that eliminates it. An attacker who can get instructions in front of the model, whether typed directly or hidden inside a retrieved document or a web page the model reads, can attempt to override its behavior. The realistic goal is containment, not a magic filter.
Containment comes from a few layers working together. Put a guardrail model or a content-safety classifier in front of and behind the main model to catch obvious injection and unsafe output. Never treat model output as trusted: OWASP's LLM05 (Improper Output Handling) exists because teams pass generated text straight into a shell, a SQL query, or a browser without sanitizing it, which reintroduces classic injection vulnerabilities through a new door. And be deliberate about agency. The moment you give a model tools that can act, book a payment, delete a record, send a message, you are in OWASP LLM06 (Excessive Agency) territory. Scope every tool to the narrowest possible permission, require human confirmation for anything irreversible, and log each tool call as a security event.
Two deployment models, side by side
The choice between a hosted API and a self-hosted open-weight model is a genuine trade, not a foregone conclusion. It comes down to who carries which burden.
DimensionHosted APISelf-hosted open-weight
Data residency
Governed by the vendor's regions and contract terms
Fully inside your boundary; no cross-border transfer
Prompt confidentiality
Prompts leave your network
Prompts never leave your control
Security burden
Model and infra security owned by vendor
You own supply chain, host, and runtime security
Supply chain risk
Low and vendor-managed
High if unmanaged; controllable with scanning and pinning
Cost profile
Per-token, scales with usage
Fixed GPU capital and operating cost
Update cadence
Automatic, sometimes without notice
You choose when and what to upgrade
Best fit
Low-sensitivity data, fast prototyping
Regulated data, sovereignty and contractual constraints
Plenty of organizations end up running both: a hosted API for low-sensitivity experimentation and a self-hosted model for anything touching regulated data. The important thing is that the decision is made per workload, on the basis of data classification, rather than by default.
Mapping it to a framework your auditors recognize
Self-hosting a model does not require you to throw out the governance you already have. It requires you to extend it. The AI-specific frameworks worth adopting are the NIST AI Risk Management Framework, including its Generative AI Profile, and ISO/IEC 42001:2023, the first management-system standard for artificial intelligence. Neither replaces your existing certifications. They sit on top of them.
In practice the mapping is direct. Your ISO 27001 information security management system already covers asset management, access control, cryptography, and supplier relationships; the model, its weights, and its data sources become new assets under that same system. NIST CSF gives you the identify, protect, detect, respond, and recover structure to slot AI-specific controls into. If you handle cardholder data, PCI-DSS segmentation requirements apply to the GPU environment exactly as they would to any other system in scope. The goal is one control framework that now includes AI, not a separate AI silo that your auditors have never seen and cannot reconcile against everything else.
A pre-go-live checklist
Before a self-hosted model serves a single production request touching regulated data, we work through the following with clients:
- Model artifact pulled from a pinned repository and revision, in safetensors format, scanned, hashed, and mirrored to an internal registry.
- Inference host hardened to a CIS Benchmark, running the container as non-root with minimal capabilities and read-only model mounts.
- GPU zone network-segmented with no unmonitored egress path, ideally air-gapped for the most sensitive workloads.
- Every caller authenticated and authorized per application, with per-client rate and token limits enforced.
- Retrieval layer honoring the source system's access controls, so users only see what they are cleared for.
- Prompt and response logging flowing into the SIEM with the correct data classification, encryption, and retention.
- Guardrail or safety model in place, and all model output sanitized before it reaches any downstream system.
- Tool and agent permissions scoped to least privilege, with human confirmation on irreversible actions.
- Model version, data sources, and controls documented as assets under your ISO 27001 and NIST CSF programs.
- Red-team pass covering prompt injection, data exfiltration, and jailbreak attempts before launch, repeated on each model upgrade.
How Aydahwa Enterprise can help
We help regulated organizations across banking, telecom, government, and critical infrastructure deploy AI on their own terms without opening a new hole in their defenses. Our work in this area sits at the intersection of three things we already do: securing infrastructure, protecting cloud and on-premises workloads, and mapping controls to the standards auditors expect. The team holds ISO/IEC 27001, PCI-DSS, SOC 2, NIST CSF, and CIS Benchmark experience, along with the Microsoft Certified: Cybersecurity Architect Expert credential, and more than 25 years of hands-on infrastructure and security work behind it.
A typical engagement starts with an architecture and threat review of the proposed deployment, moves through hardening and supply-chain controls for the model and its inference stack, and finishes with the governance mapping and red-team validation that let your risk and compliance teams sign off. If you are weighing a self-hosted model, our cybersecurity services and cloud security and migration practices cover the architecture and controls, our managed IT services team can run the environment day to day, and you can gauge your current position in a few minutes with our free cybersecurity self-assessment and readiness checklist. When you are ready to talk specifics, get in touch and we will walk through your use case.
