The pipeline choice sits underneath every SOC that works
Most security programs argue about tools. They compare SIEM vendors, benchmark EDR agents, and debate which threat-intel feed is worth the licence. Underneath all of that sits a quieter decision that decides whether any of it performs: how the telemetry actually moves. A mid-sized enterprise today generates somewhere between a few hundred gigabytes and several terabytes of security-relevant events a day once you count endpoint logs, cloud audit trails, identity events, firewall and proxy records, and application logs. That volume has to travel from where it is produced to where it is analysed, and the shape of that journey is either streaming or batch. Get it wrong and you either drown in cost or miss the alert that mattered.
We've spent years building and fixing these pipelines for banks, telecom operators, and critical-infrastructure clients, and the pattern is consistent: teams pick an architecture because it is fashionable, not because it fits the detection they are trying to run. This article lays out how to make that call deliberately, which standards constrain it, and the hybrid design we end up deploying most of the time.
What streaming actually buys you
Streaming means events are processed as they arrive, one record or one small window at a time, with no wait for a scheduled job. In a security context that translates directly into detection latency. When a service account suddenly authenticates from a new country and then enumerates a file share, the gap between the event and the alert is the gap an attacker gets to keep working. A streaming pipeline measures that gap in seconds. A nightly batch job measures it in hours.
The techniques that only work on a stream are the ones worth naming. Sessionisation, where you stitch related events into a single logical activity as they flow, needs a continuous view. Sequence detection, where the alert fires on ordering rather than a single event (failed logins, then a success, then a privilege change), depends on seeing events in near real time. Enrichment at ingest, where you attach geolocation, asset criticality, or threat-intel context before the event ever lands in storage, is far cheaper on a stream than as a retroactive join across billions of rows.
The engineering that makes streaming real
A production streaming path usually looks like this: a collector or agent ships events to a durable log such as Apache Kafka or AWS Kinesis, a stream processor like Apache Flink, Kafka Streams, or a pipeline tool such as Cribl Stream or Vector transforms and enriches them, and the output fans out to a real-time detection engine and to long-term storage at the same time. The durable log matters more than people expect. It decouples producers from consumers, absorbs traffic spikes without dropping data, and lets you replay a window of events when you deploy a new detection rule and want to test it against yesterday's traffic.
Streaming has real costs. State has to live somewhere, so windowed detections consume memory and demand careful handling of late-arriving and out-of-order events. Watermarks, exactly-once semantics, and backpressure stop being academic terms the first time a broker fills up during an incident. And a stream processor that dies silently is worse than no pipeline at all, because the dashboards stay green while detection quietly stops. Streaming rewards teams who invest in the operational discipline it demands.
Where batch still wins
Batch processing groups events and runs the analysis on a schedule or when a volume threshold is crossed. It reads as old-fashioned next to real-time dashboards, and it is still the correct answer for a large share of security work.
Threat hunting is the clearest example. A hunt asks a question across ninety days of history, joins several data sources, and often reruns the same query with variations as the hypothesis sharpens. That is a batch workload by nature, and forcing it through a streaming engine buys nothing. Compliance reporting is another. When an auditor wants proof that every privileged login over the quarter was reviewed, you are running a bounded query over a fixed window, not watching a live feed. Retrospective detection belongs here too: when a new indicator of compromise lands from an intel feed, you sweep the historical record for it, and that sweep is a batch job every time.
Batch also wins on cost and on accuracy. Processing large volumes in scheduled jobs against columnar storage is dramatically cheaper per gigabyte than keeping a stream processor and its state hot around the clock. And because a batch job sees a complete, bounded dataset, it sidesteps the entire class of correctness problems that late and out-of-order events create on a stream. For detections where a few minutes or hours of delay is acceptable, batch gives you a simpler system and a smaller bill.
Streaming and batch, side by side
DimensionStreamingBatch
Detection latency
Seconds to low minutes
Minutes to hours (schedule-bound)
Best-fit use cases
Real-time alerting, sequence and session detection, ingest-time enrichment
Threat hunting, compliance reporting, retrospective IOC sweeps, model training
Cost profile
Higher steady-state; processors and state stay hot
Lower per-GB; compute runs only when the job runs
Correctness challenge
Late and out-of-order events, windowing, exactly-once
Freshness lag; the newest events aren't analysed yet
Operational burden
High; silent failure stops detection
Lower; a failed job is easy to spot and rerun
Typical tooling
Kafka, Kinesis, Flink, Kafka Streams, Cribl, Vector
Object storage plus Spark, Presto/Trino, or the SIEM's scheduled search
The hybrid pattern we deploy most often
The framing of streaming versus batch is a false choice for anyone running a real SOC. Almost every mature pipeline we build runs both, split by the value of the data over time. Security telemetry is most valuable in its first few minutes, when a fast detection can stop an attack in progress, and it stays valuable for months as evidence for hunting, investigation, and audit. One architecture rarely serves both ends well, so we split the path.
The hot path is a stream. Every event flows through Kafka into a stream processor that enriches it, runs the latency-sensitive detections, and pushes alerts to the SOC within seconds. The same enriched event is written, in parallel, to cheap columnar storage such as Parquet files in S3 or an OpenSearch or ClickHouse cluster. That warm-and-cold store is where the batch jobs live: the nightly compliance rollups, the weekend threat hunts, the retrospective sweeps when a new indicator arrives. Security teams recognise this as the Lambda architecture, adapted so the two layers serve detection and investigation rather than duplicating a business metric.
Keeping the two paths honest
The failure mode of a split pipeline is drift. When the streaming detections and the batch queries parse fields differently, an analyst chasing a real-time alert into the historical data finds the two tell different stories, and trust in the whole system erodes fast. A shared schema fixes this. Normalising every source to the Open Cybersecurity Schema Framework (OCSF) at ingest means a detection written once reads the same whether it runs on the live stream or over a year of Parquet. Mapping detections to MITRE ATT&CK techniques and authoring them in a portable format such as Sigma pushes that consistency further, so the logic that fires in real time is the same logic you can replay across history without a rewrite.
Cost and compliance shape the design as much as latency
Two forces decide the split as firmly as detection latency does: what you are required to keep, and what you can afford to keep hot.
Retention is not optional, and the frameworks are specific. PCI DSS requires at least twelve months of log history, with the most recent ninety days immediately available for analysis. ISO 27001's Annex A control on logging expects events to be produced, protected, and reviewed, which in practice means tamper-resistant storage and a defensible review process. NIST CSF and the CIS Controls both treat centralised, retained logging as foundational rather than optional. None of these standards care whether your pipeline is streaming or batch, but all of them assume the data survives long enough to be queried, which pushes you toward a tiered store: hot storage for the recent window your analysts and streaming detections need, cheaper cold storage for the long tail the auditors need.
Cost then tells you where to draw the line between the tiers. Keeping ninety days searchable in a hot SIEM index is expensive; keeping two years of the same data in compressed columnar files in object storage costs a fraction of it. The pipeline design that survives budget review keeps only the recent, high-value window on the fast streaming-fed store and ages everything else down to batch-queryable cold storage on a defined schedule. We size that hot window to the detection and investigation needs of the team, not to a round number someone liked.
A decision checklist for your own pipeline
When we scope a telemetry pipeline with a client, the design falls out of a handful of questions asked in order:
- For each detection, what is the tolerable delay between the event and the alert? Seconds means streaming; hours means batch.
- Does the detection depend on event ordering or on a session spanning multiple events? If yes, it belongs on the stream.
- What is the required retention, and which regulation sets it? Map PCI DSS, ISO 27001, NIST CSF, and any local mandate to a concrete number of days in hot and cold tiers.
- What volume per day, at peak, must the ingest layer absorb without dropping events? Size the durable log for the spike, not the average.
- Is there a shared, normalised schema (OCSF or equivalent) so a detection reads identically on the stream and in history?
- How will you know within minutes that the streaming path has stopped? Define the heartbeat and the alert-on-silence before you go live, never after.
- What is the monthly cost of the hot tier at projected volume, and does aging data to cold storage keep it inside budget?
Answer those honestly and the architecture stops being a matter of taste. Most organisations land on a stream for the small set of latency-critical detections and batch for the larger body of hunting, reporting, and retrospective work, joined by a shared schema and a tiered store.
How Aydahwa Enterprise Can Help
Designing a security telemetry pipeline is where infrastructure architecture and cybersecurity meet, and it is work we do end to end. Our team holds credentials across both sides of that line, including ISO 27001, PCI DSS, SOC 2, and the Microsoft Cybersecurity Architect Expert certification, and we have built and operated detection pipelines in banking, telecom, and critical-national-infrastructure environments where downtime and missed detections carry real consequences.
We help organisations size and build the ingest and detection layer, normalise sources to a shared schema, map detections to MITRE ATT&CK, and set retention tiers that satisfy PCI DSS, ISO 27001, NIST CSF, and CIS Controls without overspending on hot storage. That work runs through our cybersecurity services and, when the pipeline lives in the cloud, our cloud security and migration practice. For teams that need the pipeline run day to day rather than just built, our managed IT and support services cover the operational discipline a streaming path demands.
If you want to know where your current logging and detection stands before committing to a redesign, start with our free cybersecurity self-assessment and the cybersecurity readiness checklist. When you are ready to scope the pipeline itself, get in touch and we will walk through your telemetry, your regulatory obligations, and the detection latency your team actually needs.
