Self-Hosted LLMs: Security Architecture for Running AI on Your Own Infrastructure

Sensitive Data Is Quietly Leaving Regulated Businesses Through AI Prompts

Every time an employee pastes a customer record, a contract clause, or a chunk of source code into a public AI chatbot, that data crosses a boundary that most compliance programs were never designed to police. In the banking, telecom, and critical national infrastructure engagements we run, this is now one of the fastest-growing sources of shadow data flow. The prompt looks harmless. The response is useful. And a copy of regulated information has just been transmitted to a third-party service, logged somewhere outside the organization's control, and possibly retained for model training.

The reflex answer from security teams has been to block the popular AI tools at the proxy. That rarely holds. People find workarounds because the productivity gain is real. A more durable answer is to give them the capability they want on infrastructure the business actually controls: open-weight language models, self-hosted inside your own network, air-gapped or VPC-isolated where the data classification demands it. This article covers the architecture and the security controls that make that a defensible decision rather than a new attack surface.

Why Run Your Own Models At All?

Self-hosting an LLM is more work than signing up for an API. It earns its place for a specific set of reasons, and if none of them apply to you, a well-governed commercial API with a signed data processing agreement may be the right call. The reasons that keep coming up in regulated environments are concrete.

Data sovereignty comes first. If you operate under UAE data-residency expectations, PCI-DSS scope rules, or a banking regulator's outsourcing guidance, you often need to prove where cardholder data, account data, or personal data physically lives and who can reach it. A model running on hardware you own, in a data center you can point to on a map, removes an entire category of cross-border transfer questions. Second is retention and training exposure. With your own deployment, prompts and completions never leave your logging pipeline, and no vendor clause governs whether your inputs feed a future model version. Third is availability and cost predictability at scale. Once inference volume is high and steady, the economics of owning GPUs can beat per-token pricing, and you stop being exposed to a provider's rate limits during a busy quarter.

The trade you accept in return is operational: you now own patching, model supply-chain integrity, GPU capacity planning, and the security of an inference endpoint that speaks a powerful API. That trade is manageable. It is not free.

The Three Serving Engines, and What Each One Is For

Open-weight models need a serving engine to turn model weights into an API your applications can call. Three have become the practical defaults, and they target different workloads. Picking the wrong one is a common early mistake that shows up later as either wasted GPU spend or an endpoint that falls over under real concurrency.

Ollama is the simplest. It pulls quantized GGUF model files, serves an OpenAI-compatible API, and processes requests in a straightforward queue. It runs on a workstation or a single modest GPU, which makes it ideal for a proof of concept, a developer's local environment, or an internal tool with light traffic. It is not built to squeeze maximum throughput out of a busy multi-user server.

vLLM is built for serving at volume. Its continuous batching slots new requests into an already-running batch instead of making them wait, and its PagedAttention scheme manages the key-value cache efficiently so a single GPU serves far more concurrent sessions. When you are standing up a shared internal service that hundreds of staff will hit at once, this is usually the engine that keeps latency sane.

SGLang shines when prompts overlap heavily, which is exactly the pattern that agent workflows and long multi-turn chats produce. Its prefix-aware scheduling and RadixAttention cache reuse the shared portion of prompts instead of recomputing it, so tool-calling loops and structured JSON generation run noticeably faster. If you are building AI agents against internal systems, it deserves a serious look.

Engine Comparison at a Glance

FactorOllamavLLMSGLang

Best fit

Local dev, prototyping, low-traffic internal tools

High-concurrency shared serving

Agents, tool loops, multi-turn chat, structured output

Core technique

Pre-quantized GGUF, simple queue

Continuous batching, PagedAttention

Prefix-aware scheduling, RadixAttention

Hardware floor

Laptop or single small GPU

One or more server GPUs

One or more server GPUs

Operational complexity

Low

Medium

Medium to high

Throughput under load

Modest

High

High, best on overlapping prompts

A pattern we often deploy in practice: Ollama on developer machines for experimentation, then vLLM or SGLang behind a hardened gateway for the shared production endpoint. The engine is a performance decision. The controls wrapped around it are the security decision, and that is where most self-hosting projects underinvest.

The Threat Model Changes the Moment You Self-Host

An internal inference endpoint is not a passive file server. It is a service that accepts untrusted text, executes a model against it, and in agent deployments may call other systems as a result. Treating it like just another microservice is how teams get surprised. The realistic threats break into a few groups.

The endpoint itself is a target. An OpenAI-compatible API with no authentication, reachable from the general corporate network, is an open invitation for anyone who lands on that subnet to run unlimited inference, exfiltrate whatever the model can access, or simply burn your GPU capacity. We have walked into environments where the inference port was bound to 0.0.0.0 with no key, discovered during a routine internal scan.

The model supply chain is a target. Model weights and GGUF files are downloaded artifacts, and a poisoned or backdoored model pulled from an unverified source can carry behavior you did not intend. Pinning versions, verifying checksums, and pulling only from sources you trust is basic hygiene that is frequently skipped in the rush to get something running.

Prompt injection is the threat people underestimate most. When a model reads a document, an email, or a web page as part of its context, hostile instructions embedded in that content can redirect what the model does. In an agent that can query a database or send a message, a successful injection is not a curiosity. It is a path to data exfiltration or unauthorized action. The defense is to treat model output as untrusted, keep the agent's permissions tightly scoped, and put a human or a policy check in front of any consequential action.

Finally, the logs are a target and a liability at the same time. You will want to log prompts and completions for monitoring, but those logs now contain the very sensitive data you self-hosted to protect. They need the same classification, encryption, and access control as the source systems.

A Security Architecture That Holds Up

Here is the reference shape we build toward for a regulated client running self-hosted models. It maps cleanly onto controls that auditors already recognize, which matters when the deployment lands in an ISO 27001 or PCI-DSS assessment.

Network isolation

Put the inference engine in its own segment, not on the flat corporate LAN. For the highest data classifications, run it air-gapped or in an isolated VPC with no route to the internet, so model weights are side-loaded and nothing about a prompt can leave. Every call to the endpoint travels over TLS, and mutual TLS between the application tier and the model service raises the bar further. Egress from the model segment should be denied by default and opened only to the specific internal systems an agent genuinely needs.

Authentication, authorization, and rate limits

No unauthenticated inference, ever. Front the engine with an API gateway that enforces per-application keys or short-lived tokens, applies per-identity rate limits, and rejects oversized requests. This single control neutralizes the most common finding we see. It also gives you an enforcement point for content policy and an audit trail of who called the model and when.

Model and dependency integrity

Treat models as you treat any other software artifact. Pull weights from a controlled internal registry, verify checksums, pin exact versions, and record provenance. Scan the container images that run the engine, keep the CUDA and Python dependency stack patched, and rebuild on a schedule rather than letting an image drift for a year.

Monitoring, logging, and SIEM integration

Feed gateway logs, authentication events, and anomaly signals into your SIEM so the model endpoint is visible alongside the rest of the estate. Watch for the patterns that matter: a spike in requests from one identity, prompts that look like injection or extraction attempts, and unusual egress from the agent's service account. Encrypt the prompt and completion logs, restrict them to a small group, and set a retention period that matches your data policy rather than keeping everything forever.

Data handling and secrets

Decide up front what data is allowed into a prompt for each classification tier, and enforce it at the gateway where you can. Keep API keys, database credentials, and service tokens out of prompts and out of model context entirely, held in a secrets manager and injected only into the code paths that need them.

Mapping the Controls to the Frameworks You Report Against

One reason self-hosting stalls is that security teams cannot see how it fits the compliance frameworks they already answer to. It fits well. The controls above line up with standards auditors recognize.

Control areaNIST CSFISO 27001 (Annex A)PCI-DSS v4.0CIS Controls v8

Network isolation and segmentation

PR.AC, PR.PT

A.8.20–A.8.22

Req. 1

Control 12

Authentication and access control

PR.AC

A.5.15–A.5.18, A.8.5

Req. 7, 8

Controls 5, 6

Model and software integrity

PR.DS, ID.SC

A.8.28, A.5.23

Req. 6

Controls 2, 16

Logging, monitoring, SIEM

DE.CM, DE.AE

A.8.15, A.8.16

Req. 10

Control 8

Data protection and encryption

PR.DS

A.8.24, A.8.11

Req. 3, 4

Control 3

The point of this mapping is practical. When the deployment reaches an assessment, you are not explaining a novel technology to a skeptical auditor. You are pointing at segmentation, access control, integrity verification, and logging, and showing how each requirement is met for the model service the same way it is met for any other in-scope system.

Deployment Checklist Before You Open the Endpoint

1. Classify the data that will enter prompts, and set the deployment tier (air-gapped, isolated VPC, or segmented internal) to match the highest classification allowed.
2. Choose the serving engine by workload: Ollama for dev and light internal use, vLLM for high-concurrency serving, SGLang for agent and multi-turn workloads.
3. Place the engine in a dedicated network segment with egress denied by default.
4. Require authentication on every request through an API gateway, with per-identity rate limits and request-size caps.
5. Pull model weights from a controlled source, verify checksums, and pin versions.
6. Enforce TLS in transit, and mutual TLS between the application tier and the model service.
7. Scope any agent's permissions to the minimum, and put a policy or human check in front of consequential actions.
8. Encrypt prompt and completion logs, restrict access, and set retention to your data policy.
9. Integrate gateway and auth logs into your SIEM, with alerts for injection patterns and abnormal usage.
10. Run a review against your control framework before go-live, and schedule patching and image rebuilds as a recurring task.

How Aydahwa Enterprise Can Help

We help regulated businesses in the UAE and across banking, telecom, and critical infrastructure deploy AI on infrastructure they control, without opening new gaps. Our team holds credentials that back this up, including ISO 27001, PCI-DSS, SOC 2, NIST CSF and CIS Benchmark experience, and Microsoft Cybersecurity Architect Expert certification, and we have delivered secure architecture in environments where a data leak is a regulatory event, not just an inconvenience.

A typical engagement covers the full path: classifying the data and choosing the right deployment tier, designing the network segmentation and gateway controls, hardening the serving engine and its supply chain, and wiring the whole thing into your SIEM and compliance reporting. If you want to understand your current exposure first, our cybersecurity self-assessment and readiness checklist are a fast way to see where the gaps are. You can review our cybersecurity services and cloud security and migration work for the detail, and when you are ready to scope a private LLM deployment, get in touch and we will map it to your controls from day one.