Three teams, one research base, three very different architectures
Over roughly eighteen months, DoorDash, Instacart, and Uber Eats each rebuilt search around large language models. They were chasing the same problem: understanding what a person actually means when they type "something healthy for a rainy evening" or "gf pizza" into a search bar. They read the same public research. They faced the same failure modes, from typos and synonyms to compound constraints like "vegan chicken sandwich." And they shipped three architectures that barely resemble one another.
DoorDash kept the language model mostly offline, using it to enrich a product knowledge graph in batch and to parse queries into chunks that map back to that graph, while classical retrieval still drives the runtime. Instacart put the model at the query-understanding layer, pulling company-specific context into the prompt for common searches and fine-tuning a smaller Llama-3-8B model to handle the long tail. Uber Eats went furthest, fine-tuning a Qwen model into the embedding backbone of its two-tower retrieval system, so every query and every catalogue item is represented by a vector the model produced.
The usual reading of this is an engineering story about model choice and latency budgets. That reading misses the part that matters most to anyone responsible for a production system. The decision each team really made was how deeply the model reaches into the runtime, and that single decision sets the security posture, the data-governance obligations, and the compliance surface for everything downstream. We spend a lot of time on this question in our own engagements, usually well before anyone has picked a model.
The decision underneath the architecture: how deep does the model sit?
Line the three companies up on one axis, from a model that lives at the periphery to a model wired into the critical path, and a clear spectrum appears.
DoorDash sits at the shallow end. Its language model runs offline against SKU data to extract attributes, and at request time it only segments a query into pieces that link to graph fields. The heavy lifting of retrieval stays keyword-and-graph driven. If the model went down tomorrow, search would degrade, not collapse.
Instacart sits in the middle. Common "head" queries hit an offline retrieval-augmented pipeline that is cached and latency-tolerant, while rare "tail" queries are handled by a fine-tuned model in the hot path. The model shapes what gets retrieved, but downstream retrieval is still fairly traditional.
Uber Eats sits at the deep end. The fine-tuned model is the embedding function itself. Every request depends on model-derived representations, and the model is pre-baked into every document vector in the index. This buys real capability, including cross-lingual understanding that resolves the Spanish word "pan" (bread) against the English "pan" (a cooking vessel), but it also means the model is now load-bearing infrastructure.
Each team's position traces back to what it already had. DoorDash owned a knowledge graph, so the cheapest win was making the graph richer. Instacart was drowning in separate query-understanding models, so consolidation under one strategy paid off. Uber Eats already ran two-tower embedding infrastructure per vertical, so swapping in a fine-tuned model as the shared backbone was the natural next step. The lesson the original engineering write-ups drew is a good one: ask where in your existing stack a model would actually help before asking which model to use. We would add a second question that sits right next to it. What does moving the model to that spot do to your attack surface?
Integration depth is an attack-surface decision
Every place a language model consumes attacker-controllable input is a place an attacker can try to steer it. Search queries are attacker-controllable by definition. Anyone with the app can type anything into the box. So the depth question maps almost directly onto exposure.
When the model runs offline in batch, as it largely does at DoorDash, it mostly processes your own catalogue data on your schedule. Live user input reaches it in a narrow, well-defined way, and the blast radius of a manipulated query is small because the runtime output space is constrained to concepts the graph already understands. This is the smallest surface of the three.
When the model runs at query time, as it does for Instacart's tail traffic, it now reads strings an attacker fully controls, on every rare query. That is precisely the condition described in the OWASP Top 10 for LLM Applications as LLM01, prompt injection. A crafted query is not going to hand an attacker your database, but it can push the model to emit categories or rewrites that were never intended, poison analytics, or surface items that violate the very constraints a user relies on. The risk is subtle rather than dramatic, which is what makes it easy to miss in a design review.
When the model is the embedding backbone, as at Uber Eats, it sits in the critical path of every single request. The security conversation shifts from "what can a malicious input make the model say" to "what happens to the whole platform when this model misbehaves, slows down, or becomes unavailable." A model in the hot path is a shared dependency and a potential denial-of-service amplifier. Adversarial inputs crafted to collide in embedding space, model-serving latency spikes under load, and a bad fine-tune shipped to production all become availability incidents, not just quality regressions.
None of this argues for keeping models shallow. Uber Eats got genuine value from going deep. It argues for matching the depth of your controls to the depth of your integration, and for naming the exposure out loud during design rather than discovering it during an incident.
Guardrails are security controls, not polish
The most quietly important pattern in all three architectures is the guardrail, and it is worth pulling out because engineering teams often file it under "quality" when it belongs under "security."
DoorDash's approach is the clearest example. For each query segment, an approximate-nearest-neighbour lookup pulls the top hundred taxonomy concepts from the existing graph, and the model is then told to pick from that list rather than invent labels. Retrieval-augmented generation is being used to define the entire output space, not to feed a free-form generator. The system can only ever produce concepts the rest of the design already knows how to handle.
Read that as a security engineer and it is an allow-list. It is the language-model equivalent of using parameterised queries instead of string-concatenated SQL. The model never gets to emit arbitrary output that flows unchecked into a downstream system, because the set of legal outputs is fixed in advance. Instacart's semantic-similarity filters, which drop model outputs that drift too far from the original query, are the output-validation half of the same idea.
The principle we hold teams to is simple: treat model output as untrusted input to whatever consumes it. Constrain the output vocabulary where you can. Validate against a known taxonomy or schema before the output touches a filter, a query, or a downstream service. Log the input, the raw output, and the post-validation result so you can reconstruct what happened. These controls are not a tax on capability; in the food-search cases they are part of what made the systems work at all, and DoorDash measured a roughly 30% lift in one carousel trigger rate through an architecture whose runtime stayed mostly classical and heavily constrained.
Data governance changes at every depth
Where the model sits also decides where your sensitive data ends up, and that is a compliance question with real teeth in regulated markets.
Fine-tuning, the technique behind both Instacart's tail model and Uber Eats' embedding backbone, bakes proprietary data directly into model weights. That is powerful for domain alignment and genuinely hard to govern. You cannot easily point to which records influenced which weights, you cannot cleanly honour a data-subject erasure request against a trained model, and if any personal data leaked into the training set, it now lives in an artefact that is difficult to audit. For organisations subject to the UAE Personal Data Protection Law, the GDPR right to erasure, or PCI-DSS scoping rules, a fine-tuned model is a data store that does not behave like a database.
Retrieval-augmented generation behaves very differently. The context lives in a store you own, which means you can access-control it, audit reads against it, revoke entries, and keep it inside a chosen data-residency boundary. When a regulator or an internal auditor asks where a piece of customer data is and who can reach it, "in a permissioned vector store in this region" is an answer you can defend. "Somewhere in the weights" is not.
Offline batch enrichment has its own wrinkle. If the batch job calls a hosted model API, you are shipping catalogue and behavioural data to a third party, which pulls that provider into scope for your ISO 27001 supplier controls and your data-processing agreements. The exposure is bounded and scheduled rather than per-request, but it is still exposure, and it still belongs in your third-party risk register.
The practical takeaway from this is not "fine-tuning is bad." It is that RAG, fine-tuning, and batch enrichment carry different data-governance profiles, and the right choice depends as much on your regulatory obligations and your data-classification policy as on model accuracy.
Integration depth mapped to security posture
Integration patternWhere the model sees user inputPrimary attack surfaceData-governance profileControls that matter most
Offline enrichment and batch (DoorDash-style)
Rarely and indirectly; mostly your own catalogue data
Third-party data sharing; poisoned training or catalogue data
Scheduled, bounded exposure; provider enters supplier scope
Vendor risk review, data-classification gating, output validation on the enriched graph
Query-time understanding (Instacart-style)
On every query it handles, in near real time
Prompt injection (OWASP LLM01), output drift, analytics poisoning
Mixed; fine-tuned tail model bakes data into weights, RAG head keeps it external
Constrained output vocabulary, similarity filters, input and output logging, rate limiting
Embedding backbone in the hot path (Uber Eats-style)
On every single request, in the critical path
Availability and denial of service; adversarial embedding collisions; model as single point of failure
Deep; representations of all data derive from a fine-tuned model
Capacity and failover design, model-version rollback, latency SLOs, erasure and residency strategy for training data
A checklist before you embed a model in production
These are the questions we walk clients through when a language model is about to move from a proof of concept into a live system. They are deliberately boring, because boring is what survives an audit.
- Where in the runtime will the model sit, and does anything user-controllable reach it? Draw the data flow before you write the integration.
- What is the full set of things the model is allowed to output, and is that set constrained to values a downstream system can safely handle?
- Is every model output validated against a schema or taxonomy before it reaches a filter, query, or service?
- If the model is unavailable or slow, what happens? Is there a classical fallback, and has it been tested under load?
- Does any training or fine-tuning data contain personal or regulated data, and can you honour erasure, residency, and access requirements against the resulting model?
- If you call a hosted model, which provider is now in scope for your ISO 27001, SOC 2, and data-processing obligations?
- Are prompts, raw outputs, and validated outputs logged to your SIEM with enough fidelity to investigate an incident after the fact?
- Have you threat-modelled the integration against a language-model-specific framework such as OWASP LLM Top 10 and MITRE ATLAS, rather than assuming your existing web app threat model covers it?
The infrastructure-first lesson cuts both ways
The engineering conclusion from the food-delivery teams was that existing infrastructure should drive where a model lands. That is equally true for the controls around it. A model integration does not get its own separate security programme. It should thread through the controls you already run: identity and access management for the vector store and the model-serving endpoints, network segmentation so a model service cannot reach more than it needs, secrets management for provider API keys, and centralised logging so prompts and outputs land in the same place as the rest of your telemetry.
We have seen the opposite go wrong often enough to name it. A team stands up a model-powered feature on the side, with its own keys in a config file, its own unlogged endpoint, and no fallback, and it quietly becomes the least-governed component in an otherwise well-run estate. The fix is rarely exotic. It is applying CIS Benchmarks to the hosts, mapping the integration to the NIST Cybersecurity Framework functions you already report against, and refusing to let the model be an exception to controls everything else has to meet.
How Aydahwa Enterprise can help
Most organisations we work with are not building food search. They are embedding language models into support tooling, internal knowledge systems, fraud and risk workflows, and customer-facing features, and they are hitting exactly the questions above, usually without a clear owner for the security side of the decision. That is the work we do.
With more than 25 years across UNIX, Linux, cloud, and network architecture, and certifications spanning ISO 27001, PCI-DSS, SOC 2, and Microsoft Cybersecurity Architect Expert, we bring a security-first architecture view to model integrations rather than bolting controls on after launch. Our engagements in banking, telecom, and critical national infrastructure mean we design for the regulated end of the spectrum by default, including UAE PDPL and NESA obligations.
If you are planning or reviewing a model integration, our cybersecurity services cover threat modelling, guardrail design, and secure architecture review for AI-enabled systems, while our cloud security and migration practice handles the surrounding infrastructure, segmentation, and data-residency work. Teams that need ongoing operational cover can lean on our managed IT and support services. To gauge where you stand today, start with our free cybersecurity self-assessment and the cybersecurity readiness checklist. When you are ready to talk specifics, get in touch and we will map the integration depth, the exposure it creates, and the controls that match.
