The question a regulated client keeps asking
Almost every bank, telecom, and utility we work with wants the same thing from AI right now, and they all phrase it about the same way: "Can we get this capability without our data ever leaving our own racks?" It is a fair question, and for a long time the honest answer was awkward. The models worth using lived behind someone else's API, in someone else's region, under someone else's terms of service. For an institution that classifies most of its data as confidential or restricted, that was a hard stop.
That calculus has shifted, and the reason is a training technique called knowledge distillation. It is worth understanding what it does and does not give you, because the marketing around "run AI on your own hardware" is running well ahead of the engineering reality. We have deployed and assessed enough of these systems now to have opinions grounded in what actually happens after the proof of concept.
What distillation produces, and why it is not compression
Distillation trains a small model, the student, to reproduce the behaviour of a large one, the teacher. Once training finishes the teacher steps out and the student runs alone. People often assume the student is just the teacher squeezed smaller. It is not, and the distinction matters for how you reason about risk.
Compression techniques, quantization and pruning, take one model and shrink its footprint by storing its weights at lower precision or cutting parts that contribute little. Same model, lighter. Distillation produces a genuinely separate model, with its own parameters and often a different architecture, whose only job during training was to act like the teacher. One operation shrinks a thing you already have. The other trains a new thing to imitate it. Google's Gemma family is built this way, drawing on the larger Gemini models during training, and the two ideas are frequently chained: distill first to get a capable small model, then quantize it to fit a specific device.
Soft labels and why a copy can outperform its source
Ordinary training data hands the model one answer per example. A picture of a cat is labelled "cat" and everything else is wrong. A teacher model hands over something richer: a full spread of probabilities, cat at 0.70, dog at 0.25, fox at 0.05. That spread is called a soft label, and the structure inside it, sometimes called dark knowledge, tells the student how the categories relate. The student is scored on how closely its own probability distribution matches the teacher's, so it learns the whole pattern of confidence rather than a single word. This is the mechanism behind the 2015 work that introduced the temperature control, and it is why a student can reach good performance from far fewer examples than raw labelled data would require. Each example simply carries more information.
What the results actually show
The headline results are real, and so is the fine print. In early 2025, DeepSeek used a large reasoning model to generate training examples, then fine-tuned several smaller models on them. One 7-billion-parameter student scored higher than a 32-billion-parameter model on a competition mathematics benchmark. The distilled family ran from 1.5 billion up to 70 billion parameters, and the smaller members fit on a single graphics card. That last detail is the one that changed my clients' conversations: strong performance on a narrow task became something a small team could run locally, on hardware they already owned.
The qualifier deserves equal weight. These wins show up on narrow, well-defined tasks such as mathematics and code. On broad general knowledge the same small models still trail the large ones. A distilled model can be excellent at one thing and mediocre across everything else. So when a vendor tells you their 7B model "beats" a frontier model, the claim is usually true in a narrow sense and misleading in a general one. Scope the deployment to the narrow task and it holds up. Ask it to be a general assistant and it will disappoint you.
The on-premises and air-gapped case
Here is where distillation stops being an ML curiosity and becomes an infrastructure decision. A model small enough to run on a single GPU is a model you can run inside your own security boundary. No egress to a third-party endpoint, no data crossing a border, no dependency on someone else's uptime or retention policy. For a regulated institution that is not a convenience, it is the difference between a project that clears the risk committee and one that never gets off the whiteboard.
Data that cannot leave the building
Under the UAE's Personal Data Protection Law, Federal Decree-Law No. 45 of 2021, and the sector rules that sit on top of it for banking and critical infrastructure, organisations carry real obligations about where personal data is processed and how it is transferred. An on-premises model sidesteps a large part of that problem by keeping inference inside a system you already control and already assess. In ISO 27001 terms, it lets you keep restricted-classification data within its existing controls rather than inventing a new cross-border processing arrangement and a new set of contractual safeguards to go with it. We have watched more than one AI initiative stall for a quarter because nobody wanted to own the data-transfer paperwork. Running the model locally makes that paperwork disappear.
Air-gapped environments push this further. In parts of the energy and defence work we support, the network that holds the sensitive data has no route to the internet at all. A hosted API is simply not an option there, full stop. A distilled model, delivered on physical media and stood up on an isolated GPU host, is one of the few ways to get modern language capability into those environments without punching a hole in the isolation that is the whole point of the design.
Where a small private model earns its place in security operations
The narrow-task limitation turns out to be a good match for security operations, because most of the useful work there is narrow. We have had solid results pointing small local models at a short list of well-bounded jobs: drafting first-pass detection rules from an analyst's plain-language description, summarising noisy log bundles into something a tier-1 analyst can triage quickly, clustering and de-duplicating alerts, and doing the initial read on suspected phishing so a human sees a ranked queue instead of an undifferentiated inbox. None of these ask the model to know everything. They ask it to be reliably competent at one repetitive thing, which is exactly what distillation delivers.
The newest direction makes this cheaper to build. Instead of hand-assembling a labelled dataset, the large teacher can run the whole loop: generate the training data, fine-tune the student, evaluate it against held-out examples it also generates, and repeat until the student stops improving. A 2026 study applied this to a detection task and it worked. Your team's job shrinks to defining the task and the success criteria, then checking the result against real data. One caveat carried straight through that work, and it leads directly into the part most people skip.
The supply-chain problem nobody puts on the risk register
The choice of teacher drives everything, and a distilled model can inherit far more than the task it was trained on. In a 2025 study later published in Nature, researchers took a teacher model with a particular quirk, a tendency to favour owls, and used it to generate training data made up only of number sequences. A student trained on those numbers picked up the same preference, even after the data was filtered to strip out any visible trace of it. The same effect showed up with more serious behaviours, and it only occurred when teacher and student shared the same base model. The lesson is blunt: filtering the visible training data is sometimes too coarse to catch what a model carries across, and behavioural traits can ride along in signals a human review will never see.
Sit with what that means for a security team. When you download a distilled model from a public hub, you are not just importing weights. You are importing whatever the teacher encoded, including things nobody documented and things the data filter could not remove. There is also a plain ceiling effect underneath the exotic risk: a student trained on a teacher's output tends to inherit the teacher's mistakes along with its skills, so a flawed teacher quietly hands you a flawed student. This is a software supply-chain problem wearing an AI costume, and most organisations have no control for it yet. Their software bill of materials tracks libraries and containers and stops at the model file.
Treat models as first-class assets, the way NIST CSF asks you to under Identify. That means recording provenance for every model you run, which teacher it came from, who published it, and against what benchmark it was validated. It means integrity-checking the artifact the same way you would any signed binary, and it means evaluating a new model in a sandbox against your own red-team prompts before it touches production data, rather than trusting a leaderboard score. A model you cannot trace is a model you cannot defend in an audit, and eventually someone will ask you to.
Hosted frontier API versus on-premises distilled model
Neither option is correct in the abstract. The right answer depends on your data classification, your latency budget, and how narrow the task is. This is the comparison we walk clients through.
DimensionHosted frontier APIOn-premises distilled model
Data residency
Data leaves your boundary; depends on provider region and contract
Inference stays inside your own controls; nothing egresses
Task breadth
Broad, general capability across many domains
Strong on the narrow task it was distilled for, weaker elsewhere
Cost model
Per-request, scales with usage, unpredictable at volume
Fixed hardware cost, predictable once provisioned
Latency
Network round-trip plus provider queue
Local, low and consistent, works offline and air-gapped
Provenance risk
Managed by the provider, opaque to you
Yours to verify; inherited-trait and teacher-ceiling risk is on you
Compliance fit
Requires a cross-border transfer and processing assessment
Fits existing on-prem controls and data classification
For a general-purpose internal assistant handling low-sensitivity data, the hosted API is usually the pragmatic choice. For a specific, repeatable job over restricted data in a regulated sector, the local distilled model tends to win once you price in the compliance overhead the API path drags along with it.
A checklist before you deploy a private model
- Define the one narrow task the model will do, and write the success criteria before you pick a model. Resist scope creep toward "general assistant".
- Record provenance: which teacher, which publisher, which version, validated against which benchmark. Add the model to your asset inventory.
- Integrity-check the model artifact and pin the exact version. Do not auto-pull "latest".
- Evaluate in a sandbox against your own adversarial prompts and against real samples of your data, not just a public leaderboard.
- Size the hardware honestly. A model that fits on one GPU in a demo may need headroom for concurrency in production.
- Map the data classifications the model will touch to your ISO 27001 controls and your PDPL obligations before go-live, not after.
- Keep a human in the loop for any decision with consequences. These models draft and triage well; they should not be the final word on a block, a payment, or an escalation.
- Log the model's inputs and outputs so you can audit its behaviour and catch drift over time.
How Aydahwa Enterprise Can Help
We help organisations in banking, telecom, and critical national infrastructure put AI to work without loosening the controls that keep them compliant. That covers the architecture decision of hosted versus on-premises, the secure build-out of an isolated or air-gapped inference host, and the governance wrapper around it: model provenance, integrity, sandboxed evaluation, and the mapping back to ISO 27001, PCI-DSS, NIST CSF and CIS Benchmarks. Our team holds credentials including the Microsoft Cybersecurity Architect Expert certification and 25 years of hands-on infrastructure and security work across regulated sectors, so the advice is grounded in systems we have actually run, not slideware.
If you are weighing where a private model fits, our cloud security and migration and managed IT practices can take it from design through to a supported production deployment. A good starting point is our free cybersecurity self-assessment and the cybersecurity readiness checklist, which will show you where the gaps are before you add AI to the mix. When you are ready to talk specifics, get in touch and we will scope it against your actual environment.
