A failure that refuses to spread
Picture a card payment halfway through processing. The transaction has already passed validation, enrichment, and a couple of internal service calls. Then one of those services stops answering. The customer is standing at the terminal. The clock is running, and the whole approve-or-decline decision has to land in a few hundred milliseconds.
There are two tempting moves here, and both are traps. Retry inside the same failing unit and you risk feeding a transaction into a component that is already on the floor. Hand the work to a second unit and that unit needs to know exactly how far the first one got, which means the two units now share state. Share state and you have quietly welded two independent systems into one fragile one. The failure you were trying to contain now has a bigger surface to spread across.
The way large financial platforms answer this is with cell-based architecture, and the idea travels well beyond payments. American Express engineers described their version of it recently, and it is worth studying not because most organisations run a card network, but because the underlying principle applies to any system where a local failure must not become a global one: banking cores, telecom charging systems, metering and SCADA platforms, and the regulated cloud workloads we spend most of our time designing. The interesting part is not the diagram. It is the discipline required to keep the boundaries intact when everything is under pressure.
What a cell actually is
A cell is a complete, self-sufficient copy of the processing stack. Everything a transaction needs to run lives inside one boundary: the microservices, the databases, DNS, the supporting infrastructure. That is the whole trick. If the work never has to leave the boundary to get an answer, then a problem inside the boundary has nowhere to go.
Five properties define a cell in practice:
- It deploys on its own and processes work on its own.
- It owns its services, its data stores, and its supporting components outright.
- It is a single failure domain, so a problem that starts inside stays inside.
- It can be pulled out of rotation for maintenance or during an incident while the rest of the platform keeps running.
- It has no synchronous dependency on another cell in the critical path.
That last property is the one teams get wrong. Plenty of things still cross cell boundaries: reference data replicates outward, observability data aggregates inward, routers talk to each other. What is forbidden is a blocking call to another cell while a transaction is mid-flight. The moment cell A has to wait on cell B to finish a payment, B's bad day becomes A's bad day.
Cells are not microservices, and conflating the two causes real design mistakes. Microservices split a system by function. Cells split it by failure. One cell contains many microservices. You can decompose a monolith into fifty services and still have a single blast radius, because a shared database or a shared message bus quietly re-couples everything you thought you had separated. Function boundaries and failure boundaries are different lines on the map, and you have to draw both.
Drawing the boundary is judgment, not a template
Deciding how big a cell should be is where the engineering actually happens. Draw it too large, and you have wrapped the whole enterprise in a single failure domain again with extra steps. Draw it too small, and you drown in operational overhead and cross-cell chatter.
The useful heuristic is to draw the boundary around a journey rather than around a technology. For a payment, the journey that matters is the real-time one: the minimum set of components needed to produce an immediate answer. The after-the-fact work, reconciliation, reporting, settlement enrichment, can live in a different cell entirely, because nobody is standing at a terminal waiting for it. We use the same lens when we scope resilience zones for clients. Start from the customer-facing outcome that has a hard latency or availability requirement, and pull only the components that outcome truly depends on inside the wall.
There is a cost baked into this, and honesty about it matters. Enforcing the boundary means you sometimes run duplicate copies of the same service in every cell. That duplication is not an accident to be cleaned up later. It is the price of independence, and it buys you the removal of every cross-cell network hop in the hot path. The cost does grow as the platform grows, so preventing accidental cross-cell dependencies becomes a standing discipline, not a one-time design decision.
Keeping the data where the work happens
A cell is only self-sufficient if the data it needs is already sitting inside it when a transaction arrives. That forces a decision about every piece of data the system touches, sorted by how often it changes.
Immutable data, the values that are set once and never move, is trivial: push it everywhere and forget it. Semi-static data, the exchange rates, merchant category codes, and country codes that change every few hours or few months, gets the same treatment: replicate it to every cell before any transaction asks for it. The alternative is a fall-through read, where a lookup misses the local cache and travels to a central system of record while the transaction waits. Push-and-distribute spares the first transaction from paying for a cold cache, keeps a synchronous call from leaving the cell, and moves all the replication work off the critical path. It is the inversion of the more common pull-then-cache habit, and for latency-sensitive work it is the better default.
Dynamic data, the kind that changes on every transaction, breaks that model. You cannot pre-distribute what has not happened yet, and asynchronous replication always leaves a window where a cell holds stale state. Sending a transaction to a cell with stale data adds latency at best and produces a wrong answer at worst. So the platform inverts the problem: instead of moving the data to the transaction, it moves the transaction to the data.
That is deterministic routing. The routing decision comes from the transaction's own contents, its partner, its market, its payment type, rather than from cell load or availability. Transactions that need strong consistency with each other are steered to the same cell every time. Transactions with minimal data requirements can be routed freely to whatever cell is healthy. Applying deterministic routing selectively, only where consistency demands it, keeps the routing layer from becoming a bottleneck for traffic that never needed the constraint.
The router problem: a chokepoint you can trust
Every transaction enters through a router, and any transaction moving between cells travels back through it. Cells cannot talk to each other directly, so the router is the single path between them and the single path to the outside world. When a cell finishes its work and the payment needs to reach a card issuer, the cell hands it back to the router, and the router makes the outbound call. Cells never speak past their own boundary.
This should set off an alarm for anyone who has run production systems. You have just built the thing every architecture review warns against: a component every single transaction depends on. So the question is unavoidable. How do you keep that chokepoint from becoming the platform's weakest point?
The answer is restraint, applied in two directions. First, keep the router deliberately dumb. It gets just enough message-parsing ability to pull out a value and route on it, and nothing more. It does not know what those values mean. The instant a router starts accumulating business rules, those rules need data, the data needs lookups, and your thin router has grown into exactly the centralised system the architecture was built to avoid.
Second, starve it of dependencies. The closer a component sits to the edge, the fewer things it should rely on. Router state lives in non-persistent stores so instances stay as close to stateless as the design allows. Logging runs through an asynchronous logger that drops records rather than block a transaction when its buffer fills. Configuration loads into memory and refreshes in the background, so an unreachable config service just leaves the router running on its last known good values. Instances run in parallel across regions, active-active rather than active-standby, because active-standby only earns its keep when you have state to manage, and the whole point of this component is that it has almost none.
The lesson generalises past payments. Some chokepoints are structural. The architecture needs one place where cross-boundary decisions happen, and pretending otherwise just hides the dependency. The honest move is to make that one place simple enough to trust, and then spend your engineering budget keeping it that way.
When a service fails mid-flight
Back to the transaction that stalled at the start. Inside a cell, an orchestrator service runs the workflow, calls the other services in turn, and watches their health continuously. When a required service fails, or when too few instances of it are left running, the orchestrator does something that looks wasteful and turns out to be wise.
It halts the transaction, hands it back to the router, and the router picks a healthy cell. Processing restarts there from the beginning, using the original input. Every service call the failing cell had already completed is thrown away. The second cell does not resume where the first left off. It starts over.
The reasoning goes straight back to the boundary. Resuming would require the second cell to read state from the first, and that link is precisely the shared state the whole design exists to prevent. Discarding a few hundred milliseconds of completed work is cheap. A permanent structural dependency between every pair of cells is not. For a workload measured in milliseconds, restart-in-place is close to free. For a workload where a single unit of work runs for minutes, the maths flips and checkpointing starts to look attractive, which is exactly the kind of trade-off worth reasoning through explicitly rather than inheriting from a reference architecture.
Restarting elsewhere only works up to a point, and choosing where that point sits is a real design decision. Think of it as a point of no return. Rerouting is safe while the transaction is still inside the platform. Once it has gone out to an external system, an issuer, a partner network, a downstream ledger, it stays where it is. Card authorisations are deliberately structured so that irreversible step falls as late as possible, which makes the recoverable window as wide as it can be. You want to design your own flows the same way: push the commit point as far toward the end as the business logic allows, so the largest possible share of a transaction's life remains survivable.
Where late rerouting is not possible, safety comes from idempotency. Each transaction carries a unique identifier that stays constant across every retry and reroute, and downstream systems use it to suppress duplicates. Failback is paced with canary percentages rather than flipped all at once, so a recovering cell is validated under partial load before it takes full traffic. Speed, idempotency, and paced recovery together close the window in which a recovered cell might write stale state after its transactions already completed somewhere else.
Why a resilience pattern is also a security pattern
Here is the part that gets underplayed in the reliability write-ups, and the part we care about most in our security engagements. A failure domain and a security blast radius are the same shape drawn with a different pen.
Everything that keeps a service outage inside one cell also works to keep an intrusion inside one cell. A cell with no synchronous dependency on its neighbours, its own data stores, and a single narrow, business-logic-free path in and out is, almost incidentally, a well-segmented security zone. An attacker who compromises a service inside one cell finds the same walls a failing service finds: no direct path to the next cell, no shared database to pivot through, no lateral route that does not pass through a deliberately dumb, heavily scrutinised router. That is network segmentation and least-privilege lateral movement control, arrived at from the resilience side of the house.
This is why the pattern lines up so cleanly with the frameworks regulated clients answer to. PCI-DSS has always pushed segmentation to shrink the cardholder data environment; cell boundaries are segmentation with teeth. ISO 27001's continuity controls and the NIST CSF Recover function both ask you to demonstrate that you can lose a piece of the estate and keep operating; a cell you can pull out of rotation is that demonstration made concrete. For financial entities, the EU's Digital Operational Resilience Act now makes operational resilience a supervised obligation rather than a good intention, and for critical infrastructure operators, NIS2 raises the same bar. In every one of those regimes, the auditable question is the same: when one part breaks or is breached, what else goes with it? An architecture whose honest answer is "one cell's worth of traffic, and here is how we drain it" is in a very different position from one whose answer is "we are not sure."
How the containment approaches compare
ApproachBlast radius of a single failureCross-component couplingOperational complexityBest suited to
Monolith
Entire platform
High, everything shares a process and usually a database
Low to start, high once it grows
Small, early-stage systems with modest availability needs
Microservices, shared data plane
Often platform-wide via shared stores and buses
Hidden but high, re-coupled through shared infrastructure
Moderate
Teams optimising for delivery speed and functional ownership
Multi-region active-passive
Region, with a failover gap
Moderate, standby carries state
High, failover is rarely exercised
Disaster recovery against whole-region loss
Cell-based architecture
One cell's share of traffic
Low by design, no synchronous cross-cell calls
High, but the complexity buys containment
Regulated, high-availability, security-sensitive workloads
None of these is a free lunch, and the last row is the least free of all. Cells trade management overhead and genuine complexity for a smaller failure impact. They also change the shape of failure rather than its frequency: run more independent units and you will see more individual failures, each one smaller. That is the deal. You are buying a world where more things fail but fewer of them matter.
Drawing your first cell boundary
If you are looking at a critical system and wondering where to start, this is the sequence we walk clients through:
- Name the journey with the hardest requirement. Find the customer-facing outcome that has a real latency or availability constraint, and treat that journey as the thing you are protecting.
- Pull only what that journey needs inside the boundary. Services, data, DNS, supporting infrastructure. If the outcome does not depend on it in real time, leave it out.
- Classify every piece of data the journey touches by how often it changes, and decide push-and-distribute or move-the-work-to-the-data for each, before you write any routing logic.
- Hunt down every synchronous cross-boundary call in the hot path and remove it. This is the hard part, and it is the whole point.
- Design the entry point to be dumb. If your router needs business data to make a routing decision, the boundary is in the wrong place.
- Choose your point of no return deliberately, and push it as late in the flow as the business rules allow. Where you cannot, make every step idempotent.
- Rehearse pulling a cell out of rotation. A failure domain you have never drained under load is a hypothesis, not a control.
How Aydahwa Enterprise can help
Resilience and security are the same conversation held twice, and cell-based thinking is where the two meet. We work with banking, telecom, and critical-infrastructure clients to draw failure and security boundaries that hold up under real incidents, not just on the architecture diagram. That means mapping the journeys that carry your hard availability requirements, finding the shared state and synchronous cross-boundary calls that quietly re-couple systems you believed were independent, and aligning the result with the frameworks you answer to, whether that is PCI-DSS, ISO 27001, the NIST CSF, DORA, or NIS2.
If you want to know where your current architecture would spread a failure or an intrusion today, our free cyber security self-assessment and readiness checklist are a fast first read. For deeper work, our cybersecurity services cover segmentation, containment, and resilience engineering, while our cloud services team handles the migration and multi-region design that cell-based patterns depend on, and our managed IT support keeps the boundaries intact once they are in place. When you are ready to pressure-test how your systems fail, get in touch.
