The Real Cost of Running LLMs On-Premises for Regulated Industries

The question a bank CFO asked me last quarter

A retail bank we work with wanted an internal assistant. Nothing exotic. Staff would ask it questions about policy documents, draft customer letters, and summarise long compliance memos. The catch was the one every regulated business runs into: none of that material could leave the bank's own network. Customer records, transaction notes, and internal risk assessments cannot be posted to a public API sitting in another country, full stop. So the assistant had to run on their own hardware, inside their own data centre.

Then their finance lead priced the GPUs and called me. The number was far higher than anyone had sketched on the business case, and it kept climbing as they added more concurrent users. He wanted to know whether he was being quoted honestly. He was. What nobody had explained to him was where the money in a self-hosted language model actually goes, and it is almost never where people expect.

If your organisation handles regulated data in the UAE, and most of our clients in banking, telecom, and critical national infrastructure do, you will face this same decision. This article walks through the engineering reality behind the cost, the levers that bring it down, and the security and compliance reasons that make on-premises AI worth the trouble anyway.

Where the money actually goes

People assume the expensive part of running a large model is loading its weights. That matters, but it is a fixed, one-time cost. A 70-billion-parameter model in half precision occupies roughly 140GB before it serves a single request, and that number does not change no matter how busy the box gets.

The cost that grows, the one that surprised the bank, comes from a block of working memory called the KV cache. When a model generates a reply, it stores a key vector and a value vector for every token it has already seen. It does this so it never has to recompute earlier work. The technique is sound engineering. The problem is that this cache lives on the same scarce GPU memory as the model, and it expands with every token in the conversation and every user you serve at once.

Here is the figure that reframes the whole conversation. For a Llama 3 70B model at a context length of 128,000 tokens, the KV cache for a single request comes to roughly 40GB. One long conversation can fill half of an 80GB accelerator on its own, before a second user has even logged in. That is why the finance lead's cost curve bent upward the moment his team modelled real concurrency.

Why a long prompt costs more than a short one

Token generation happens in two phases, and they wear on the hardware differently. The first is prefill, where the model reads your entire prompt in one parallel pass and builds the cache. Prefill hammers the arithmetic units, so we call it compute-bound. The second is decoding, where the model emits the answer one token at a time. For each new token it has to read every key and value already sitting in the cache. That read repeats for the whole output.

This is the part worth internalising. The expense of a long context is less about holding the cache and more about sweeping through all of it on every single token you produce. A bigger cache means more data crossing the memory bus per token, which shows up directly as slower, costlier responses. It also explains a class of out-of-memory errors that look bizarre at first, where the model fits comfortably but a request still falls over. The weights fit. The cache for that request did not.

The scaling math worth keeping on a napkin

Before anyone signs a purchase order, they should understand how the cache scales, because it is not complicated and it drives the entire capacity plan. The size of the cache is the product of a short list of numbers:

• A factor of two, one for the key and one for the value.
• The number of layers, because each layer keeps its own cache.
• The number of key-value heads per layer.
• The head dimension, the length of each stored vector.
• The bytes used to store each number.
• The context length in tokens, one entry each.
• The batch size, meaning how many requests you serve at once.

Two of those terms deserve a hard look. The cache grows in a straight line with the token count, so doubling the context doubles the memory. It grows the same way with batch size, so serving twice as many users at once doubles it again. Concurrency and context length are not free features you switch on later. They are the two dials that decide how many GPUs you buy.

Put real numbers in and it becomes concrete. Llama 3 70B has 80 layers, 8 key-value heads, a head dimension of 128, and stores each number in 2 bytes. Run that out at 128,000 tokens for one request and you land near 40GB. That single line of arithmetic is what a capacity plan for on-premises AI is built on, and it is exactly the calculation most vendor quotes gloss over.

Techniques that pull the cost down

The good news is that every part of that equation has a matching optimisation, and the open-source serving stack has matured fast. Some of these choices are baked in when a model is trained, so you inherit them with your model selection. Others are things you configure at serving time. A competent build uses several together.

TechniqueWhat it targetsPractical trade-off

Grouped-query attention (GQA)

Cuts the number of key-value heads, shrinking every token's footprint

Decided at training time; you get it by choosing a model built with it, such as Llama 3

Multi-head latent attention (MLA)

Compresses keys and values into a smaller latent form, as used in DeepSeek-V2 and V3

Also a model-architecture choice, not a serving toggle

Quantisation (KV cache and weights)

Stores each number in fewer bits, for example 8-bit instead of 16-bit

Roughly halves memory; needs validation so accuracy on your tasks does not slip

Cache eviction and attention sinks

Keeps fewer tokens in the cache for very long or streaming sessions

Trades some long-range recall for a smaller, steadier footprint

PagedAttention (vLLM)

Manages cache memory in pages instead of one contiguous block, cutting waste

Mature and widely deployed; the practical default for self-hosted serving

Prefix caching

Reuses the cache for shared prompt prefixes across requests

Large wins when many requests share the same system prompt or document

The pattern across all of them is the same. Decoding reads the whole cache on every token, so the cache is a bandwidth cost as much as a storage one, and anything that makes it smaller makes the system faster and cheaper at the same time. In our engagements the combination that does the most work for the least risk is a GQA-based model served through vLLM with PagedAttention, 8-bit KV quantisation after task validation, and prefix caching for the shared system prompt. That stack routinely turns an unaffordable quote into a sensible one.

The security and compliance case for keeping it in-house

All of that cost is why teams look longingly at a public API, where someone else owns the GPUs. For a marketing chatbot on a public website, that is often the right answer. For regulated data, it usually is not, and the reasons are the same ones that shape every other part of a regulated IT estate.

Start with data sovereignty. Under the UAE Personal Data Protection Law, Federal Decree-Law No. 45 of 2021, and the sector rules layered on top of it for banking and telecom, you are accountable for where personal data goes and who can reach it. Sending customer records to a model hosted outside your control, and outside the country, is a cross-border transfer you then have to justify. Running the model inside your own boundary removes that entire question. For the most sensitive workloads, some of our clients go further and run the stack air-gapped, with no route to the public internet at all.

Then there is the certification burden that regulated firms already carry. An on-premises AI service is not a special exception to your ISO 27001 information security management system, your PCI-DSS scope if it touches cardholder data, or your SOC 2 commitments. It is another system inside them, and it has to be treated that way. The host operating system gets hardened against a recognised baseline such as the CIS Benchmarks. Access to the inference servers is controlled, logged, and reviewed. The whole service is mapped against a framework such as the NIST Cybersecurity Framework so you can show identify, protect, detect, respond, and recover for it like anything else.

The model itself introduces risks that traditional infrastructure does not. Prompt injection, where hostile text hidden in a document steers the model into ignoring its instructions, is now a mainstream attack, and it sits at the top of the OWASP list for large language model applications. Data exfiltration through the model, where a user coaxes it into revealing training data or another tenant's context, is a real concern when you serve many teams from one deployment. These need controls of their own: input and output filtering, strict tenant isolation, guardrails on what the model is allowed to retrieve, and monitoring that actually understands the traffic. Self-hosting does not hand you these controls automatically. It hands you the ability to build them, which a public API never will.

Finally, the serving stack is software, and it ships on the same fast release cycle as the rest of the open-source AI world. vLLM, the model weights, the CUDA layer, and the surrounding Python dependencies all need patching, pinning, and a proper software supply-chain check. This is ordinary DevSecOps applied to a new kind of workload, and skipping it is how a self-hosted model quietly becomes the softest target in the building.

Cloud API versus self-hosted, without the sales pitch

Neither option is universally right. The honest comparison looks like this.

ConsiderationPublic cloud APISelf-hosted on-premises

Data sovereignty

Data leaves your boundary; cross-border transfer to justify

Data stays inside your control, air-gap possible

Upfront cost

Minimal; pay per token

Significant GPU capital or reserved capacity

Cost at scale

Grows with usage, can become large and unpredictable

Fixed once sized; predictable per year

Compliance fit for regulated data

Hard; depends on the provider's terms and location

Strong; the workload lives in your existing controls

Operational burden

Low; the provider runs the infrastructure

Higher; you own capacity, patching, and uptime

Model choice and control

Limited to what the provider offers

Full; any open-weight model you can host

The dividing line we use with clients is straightforward. If the data is public or low sensitivity and volumes are modest, a public API is faster and cheaper to start with. If the data is regulated, the volumes are steady and high, and predictable cost matters to the business, self-hosting usually wins over a two to three year horizon, and it is often the only option compliance will sign off on.

A checklist before you buy a single GPU

When we scope an on-premises AI build for a regulated client, we work through this list before any hardware is ordered. It saves the awkward call the bank's finance lead had to make.

1. Classify the data the model will touch, and confirm with your DPO or compliance team whether it can legally leave the country. That answer alone often decides the architecture.
2. Estimate real concurrency and context length, then run the KV cache equation for your chosen model. Size GPUs for the peak, not the average.
3. Pick a model built with GQA or MLA so you inherit a smaller cache from the start.
4. Plan the serving stack around vLLM with PagedAttention, and budget the accuracy testing for KV and weight quantisation rather than assuming it is free.
5. Bring the deployment into your ISO 27001 scope, harden the host to a CIS Benchmark, and map the service to the NIST CSF.
6. Design against the OWASP Top 10 for LLM applications: input and output filtering, tenant isolation, retrieval guardrails, and logging that a SOC can actually use.
7. Put the whole stack under patch management and software supply-chain control, the same as any other production system.
8. Decide up front whether the workload needs a full air-gap, and design the network so that choice is enforced, not merely intended.

How Aydahwa Enterprise Can Help

We spend most of our time on exactly this kind of problem: helping regulated organisations in the UAE and the wider region adopt new technology without breaking the compliance posture they have worked hard to build. For on-premises AI that means sizing the infrastructure honestly, choosing and tuning the serving stack, and wrapping it in the controls that let your auditors sign off.

If you are weighing a self-hosted model, our cybersecurity services cover the threat modelling, hardening, and monitoring the deployment needs, while our cloud and infrastructure practice handles the capacity planning and the build itself, whether it lands on-premises, in a private cloud, or air-gapped. Day-to-day operation and patching can sit with our managed IT support team so the stack stays current instead of drifting into risk.

If you would rather start by understanding where you stand, our free cybersecurity self-assessment and readiness checklist are a quick way to gauge whether your current controls would stretch to cover an AI workload. When you are ready to talk specifics, get in touch and we will walk through your data classification and a realistic capacity plan before anyone quotes you a GPU bill.