Why open-weight models are now a board-level infrastructure decision
A few years ago, using a large language model meant sending your data to someone else's API and trusting their word about what happened to it afterward. That is no longer the only option. Open-weight models such as Llama, DeepSeek, Mistral, Qwen, and Gemma now ship with downloadable weights that run on hardware you control, and their quality has closed enough of the gap with hosted frontier models that self-hosting is a serious choice rather than a hobbyist experiment.
For a bank in Abu Dhabi, a telecom operator, or an operator of critical national infrastructure, that shift matters for one reason above all others: the data never has to leave the building. In our engagements across regulated sectors we keep meeting the same wall, where legal and compliance teams block a promising AI use case because customer records, transaction data, or operational telemetry cannot be shipped to a third-party endpoint in another jurisdiction. Open-weight models remove that wall. They also hand you the entire security burden that the API provider used to carry, and most teams underestimate how heavy that burden is.
This article walks through what running open-weight models on your own infrastructure actually involves from a security and compliance standpoint, where the real risks sit, and how to build a deployment that a regulator, an auditor, and your own SOC can all live with.
What "open-weight" means, and what it does not
The terminology causes more confusion than almost anything else in this space, so it is worth being precise. An open-weight model is one where the trained parameters, the weights, are published for download. You can run inference locally, fine-tune, quantise, and deploy without asking permission for each call. That is different from fully open-source, where the training data and training code are also released. Most of the popular models are open-weight but not fully open source: you get the artifact, not the recipe that produced it.
Licensing follows the same pattern and deserves a careful read before anything reaches production. DeepSeek's models are distributed under permissive MIT terms. Mistral and several Qwen releases use Apache 2.0. Llama ships under Meta's community license, which is permissive for most commercial use but carries an acceptable-use policy and a monthly-active-user threshold that a large enterprise can actually cross. Gemma has its own usage terms. We have reviewed procurement decisions where a team assumed "open" meant "do whatever you want" and later discovered a clause that changed the risk calculus. Treat model licensing the way you would treat any other third-party software license: read it, record it in your software bill of materials, and get sign-off.
The security-relevant point is this. When you download weights, you are importing a large binary artifact of unknown internal composition into your trusted network. It behaves less like source code you can audit line by line and more like a compiled dependency you have to trust and contain. That framing should guide every control you put around it.
The threat model changes the moment the model runs in your data centre
With a hosted API, the provider owns the model, the inference servers, the GPUs, the patching, and the physical security. Your responsibility stops at the network boundary and the data you send. Self-hosting inverts that. You now own the full stack, and each layer brings its own attack surface.
Model provenance is the first concern. Weights pulled from a public hub can be tampered with, and the legacy Python pickle serialization format used by some checkpoints can execute arbitrary code at load time. This is not theoretical. Malicious models carrying embedded payloads have been found on public repositories. The mitigation is straightforward once you know to look for it: prefer the safetensors format, which cannot carry executable code, verify checksums against the publisher, and scan every artifact before it touches a GPU host.
The inference stack is the second concern. Serving frameworks like vLLM, Hugging Face Text Generation Inference, Ollama, and llama.cpp are fast-moving open-source projects, and like any such software they ship CVEs. An inference server exposed on the wrong interface, running as root, with a known deserialization flaw, is a genuine remote-code-execution path straight into your GPU fleet. Treat these servers as internet-facing application infrastructure even when they are not internet-facing.
The application layer is the third, and it is where most incidents will actually happen. Prompt injection, insecure handling of model output, over-permissioned tool and function calling, and data leakage through verbose logs are the everyday failure modes. The OWASP Top 10 for LLM Applications is the most useful reference here because it names these categories concretely and maps cleanly to controls a security team already understands.
Hosted API versus self-hosted open-weight: an honest comparison
Self-hosting is not automatically more secure, and pretending otherwise sets teams up for failure. It moves risk rather than eliminating it. The table below reflects the trade-offs we walk clients through before they commit budget.
DimensionHosted frontier APISelf-hosted open-weight
Data residency
Data leaves your environment; depends on provider region and contract
Data never leaves your network; full residency control
Patching and CVE burden
Provider's responsibility
Yours, across the model, serving stack, drivers, and OS
Attack surface
Narrow: API key and transport
Broad: full inference and GPU stack you must harden
Cost model
Per-token, scales with usage, predictable to forecast
Capital plus operations for GPUs; cheaper at sustained high volume
Compliance evidence
Inherited from provider certifications and your DPA
You produce the evidence; easier to map to on-prem controls you already run
Air-gap capability
Not possible
Fully achievable
Model control
Provider can deprecate or change the model under you
You pin a version and keep it as long as you need
The pattern we recommend most often is not one or the other. Route low-sensitivity, high-volume workloads to whichever option is cheaper, and keep regulated data on a self-hosted model behind your own controls. The decision should be driven by data classification, not by which technology is fashionable this quarter.
A reference architecture for on-premises and air-gapped inference
The design below is what we deploy for clients who need open-weight inference inside a regulated boundary. It assumes you already run a mature on-prem or private-cloud estate, which most infrastructure teams in banking and telecom do.
Network segmentation and egress control
Place the GPU inference nodes in a dedicated, tightly segmented zone with no outbound internet access. Models and dependencies enter through a controlled artifact pipeline, never by pulling directly from a public hub at runtime. For a true air-gapped build, the model registry, container images, and Python packages are mirrored internally and the inference zone talks only to an internal gateway. This single control eliminates a large share of exfiltration and supply-chain risk before you write a line of application code.
A hardened inference tier
Run the serving framework in containers as a non-root user, with read-only filesystems where possible, resource limits, and seccomp profiles. Apply CIS Benchmarks to the host OS and to the container runtime or Kubernetes layer. Keep the serving software on a defined patch cadence and subscribe to its security advisories, because a quiet CVE in vLLM or TGI is not something you want to learn about from an attacker.
An access gateway in front of the model
Never let applications talk to the raw inference endpoint. Put a gateway in between that enforces authentication and authorization, applies per-application rate limits, strips or redacts sensitive fields, and writes an audit trail of who asked what. This is also the right place for input and output filtering, so that prompt-injection attempts and unsafe responses are inspected at a chokepoint you own rather than scattered across every calling service.
Logging that helps compliance without creating a new data-leak
Prompts and responses can contain the very data you were trying to protect. Log enough to investigate an incident and satisfy an auditor, then classify and protect those logs at the same level as the source data. We have seen well-meaning observability setups quietly copy regulated customer data into a logging cluster with weaker controls than the system it came from. That is a finding waiting to happen.
GPU and multi-tenancy hygiene
If multiple teams share a GPU cluster, isolate their workloads. GPU memory is not automatically wiped between tenants, and residual data in device memory is a real cross-tenant concern. Enforce separation at the scheduling layer and validate it, rather than assuming the platform handles it for you.
A hardening checklist you can hand to an engineer
- Download weights only in safetensors format; verify publisher checksums and signatures before use.
- Scan every model artifact and container image for malware and known vulnerabilities in the pipeline, before deployment.
- Record each model, its version, and its license in your SBOM and asset inventory.
- Run inference servers as non-root, in isolated network segments, with no outbound internet from the GPU zone.
- Mirror models, images, and packages internally so production never pulls from a public source at runtime.
- Front every model with an authenticated, rate-limited gateway that logs and filters requests and responses.
- Apply CIS Benchmarks to hosts, container runtime, and orchestration; patch the serving stack on a tracked cadence.
- Classify and protect prompt and response logs at the same sensitivity level as the underlying data.
- Constrain tool and function calling with least privilege; a model should never hold broad credentials.
- Red-team the deployment for prompt injection and output-handling flaws before it goes live, and again after major changes.
Mapping the deployment to standards auditors recognise
A self-hosted model does not sit outside your existing compliance program. It sits squarely inside it, which is one of its quiet advantages. The controls above map directly onto frameworks your organisation is likely already measured against.
Under ISO 27001, the model becomes an information asset with an owner, a classification, and access controls, and the supply-chain checks feed straight into your supplier and acquisition controls. The NIST Cybersecurity Framework gives you a clean way to structure the work across Identify, Protect, Detect, Respond, and Recover, and the newer NIST AI Risk Management Framework extends that into model-specific risks such as adversarial attacks and misuse. For payment environments, PCI-DSS segmentation and logging requirements apply to any inference tier that can touch cardholder data, which is a strong argument for keeping that tier air-gapped from the model entirely. CIS Benchmarks give you concrete, testable hardening baselines for the hosts and orchestration underneath.
For organisations in the UAE, data-residency expectations add weight to the on-premises case. Keeping inference inside the country, and inside your own controls, is far simpler to evidence than negotiating cross-border data-processing terms with a foreign provider. That single fact is often what turns an AI project from blocked to approved.
Where teams get this wrong
Two failure patterns come up repeatedly. The first is treating the model as an application feature rather than as infrastructure, so it gets deployed by a project team without the network segmentation, patching discipline, or logging controls that any other production system would demand. The second is the opposite: security teams block open-weight adoption entirely because it looks unfamiliar, pushing frustrated business units toward shadow use of public APIs with real customer data, which is a far worse outcome than a properly governed on-prem deployment.
The productive path runs between those two. Open-weight inference is ordinary infrastructure with a few new characteristics, and it responds well to the controls you already run. The work is in applying them deliberately rather than assuming the model is either harmless or unmanageable.
How Aydahwa Enterprise can help
We help organisations in banking, telecom, and critical national infrastructure adopt AI without loosening their security or compliance posture. Our team holds credentials including ISO 27001, PCI-DSS, and the Microsoft Cybersecurity Architect Expert certification, and we build to NIST CSF and CIS Benchmark baselines by default, so an open-weight deployment lands inside a governance model your auditors already trust.
A typical engagement starts with data classification and a threat model for the intended use case, moves through a hardened reference architecture for on-premises or air-gapped inference, and finishes with the evidence pack your compliance team needs. If you are weighing self-hosted models against hosted APIs, or you already have an inference deployment that has not been through a security review, our cybersecurity services and cloud security and migration teams can help you design and validate it, and our managed IT services can keep it patched and monitored afterward.
To gauge where you stand today, run our free cyber security self-assessment or work through the cybersecurity readiness checklist. When you are ready to talk specifics, get in touch and we will scope it with you.
