Your logs are the only witness to most security incidents
A digital bank that pushes a trillion log lines a day is no longer an exotic case study. Mid-market companies we work with across the UAE now move tens of billions of events a week through their pipelines during normal operations, and the uncomfortable truth is that those logs are usually the only record that could ever prove, or disprove, that a breach happened. When one of the largest digital banks in the world recently tore out its third-party logging vendor and rebuilt the whole platform in-house, the industry wrote it up as an observability and cost story. From where we sit, having run security operations for banks and telecom operators, it was a security and compliance decision wearing an engineering costume.
The bank's problem will sound familiar to anyone who has stood up a Security Operations Centre. Costs were climbing with no ceiling in sight. The team had almost no visibility into how logs were collected or stored, so when an incident hit they could not always trust that the evidence they needed would actually be there. Ingestion spikes slowed down queries at exactly the moment engineers were trying to investigate a live problem. Dashboards and alerts were welded to the vendor, so switching was close to impossible. Every one of those is an operational headache. Every one is also a detection gap, an audit finding waiting to be written, or both.
This article is about how to build a log pipeline that holds up under real load, keeps your costs predictable, and gives your SOC and your auditors the evidence they actually need. The engineering patterns are well understood. What gets missed is that the same design choices that make a pipeline fast and cheap are the ones that make it defensible.
Logging is a security control, not a convenience
Every serious framework treats logs as a first-class control, and the ones that regulate banking and payments treat them as non-negotiable. If you handle cardholder data, PCI DSS v4.0 Requirement 10 spells out what you must record for every access to system components and cardholder data, requires time synchronisation across systems, and sets a minimum of one year of retention with at least the most recent three months immediately available for analysis. ISO 27001:2022 folds event logging, clock synchronisation, and protection of log information into controls A.8.15, A.8.16, and A.8.17. NIST Cybersecurity Framework 2.0 puts logging at the centre of the Detect function under Adverse Event Analysis, and CIS Controls v8 makes Control 8, Audit Log Management, one of the foundational eleven.
Read those together and a pattern emerges. Regulators do not just want you to collect logs. They want the logs to be complete, tamper-resistant, time-accurate, retained for a defined period, and retrievable fast enough to be useful during an investigation. A pipeline that drops events during a traffic spike fails the completeness test. One that lets any administrator quietly edit stored logs fails the integrity test. One that takes six hours to return a query fails the "available for analysis" test. The architecture is the compliance posture. You cannot bolt evidentiary quality on afterward.
There is a harder truth underneath the checklists. In a real intrusion, the attacker's actions leave traces only in the telemetry you happened to be collecting, in the format you happened to keep, for as long as you happened to retain it. We have investigated incidents where the initial access was three weeks old and the only reason we could reconstruct the kill chain was a 45-day retention window on authentication logs. We have also walked into engagements where the answer to "what did the attacker touch" was simply unknowable, because the logs that would have told us had already rolled off a seven-day retention tier chosen to save money. Retention is not a storage setting. It is the length of the memory your security programme is allowed to have.
Where log pipelines break under load
Most logging platforms are fine on the day they are built and start failing eighteen months later, when volume has quietly grown fivefold. The failure modes are consistent, and they matter for security specifically.
Ingestion and query share the same fate
The single most common design flaw is coupling the path that receives logs to the path that searches them. When both run on the same cluster, a burst of inbound traffic, which is exactly what a denial-of-service attempt or a misbehaving deployment produces, steals the resources your analysts need to query. The moment you most need to investigate is the moment your search slows to a crawl. Decoupling the two is the first thing we look for, and the first thing many teams have not done.
Cost scales faster than value
Log volume grows with your estate, but the value of any individual log line does not. Debug chatter from a chatty microservice costs the same to ingest and store as a privileged-access event, yet one is noise and the other is gold. Platforms that ingest everything at a flat per-gigabyte rate turn security teams into cost centres and create pressure to drop retention or sampling in ways that quietly blind the SOC.
Blind spots in the pipeline itself
If you cannot see whether your log forwarder is dropping events, you are trusting a black box with your only source of truth. Pipelines need their own telemetry: ingestion rates, buffer depth, drop counts, end-to-end latency. Without it, a silent forwarder failure looks identical to a quiet day, and you find out during the post-incident review that three days of logs were never collected.
Vendor gravity
When alerting logic, dashboards, and detection rules live inside a proprietary platform, the switching cost is not the data migration. It is rebuilding years of institutional detection knowledge. That lock-in is a business risk, and during contract renewals it becomes a budget risk that lands on the security team's desk.
The architecture that holds up
The shape of a resilient log platform is now fairly settled, whether you assemble it from open-source parts or buy a managed version. The principles are what carry over.
Separate ingestion from storage and query. Treat log collection as a streaming problem. Lightweight forwarders such as Fluent Bit, Vector, or the OpenTelemetry Collector run close to the workloads and ship events into a buffer, often a Kafka or Redpanda topic, that absorbs spikes. A processing layer then filters, normalises, and enriches events before they ever hit storage. Because ingestion is decoupled, a flood of inbound data fills the buffer instead of knocking over your search cluster.
Filter and enrich in flight. This is where security value is created or destroyed. Drop the debug noise, but never the audit-relevant events. Normalise fields into a consistent schema so a source IP is always called the same thing regardless of which system emitted it. Enrich with context an analyst would otherwise have to look up by hand: geolocation, asset criticality, user role, threat-intelligence matches. A normalised, enriched event is one a detection rule can actually fire on. A raw, inconsistent one is a line an analyst has to interpret at 3 a.m.
Store in tiers, and store cheaply. Hot data that analysts query daily belongs in a fast search engine such as OpenSearch, Elasticsearch, or Loki. Older data that you must retain for compliance but rarely touch belongs in object storage such as Amazon S3 or Azure Blob, written in a columnar format like Parquet that compresses aggressively and scans efficiently. A distributed SQL engine such as Trino or Amazon Athena can query that cold tier directly when an auditor or an incident demands it. This tiering is what lets you keep a year of logs for PCI DSS without paying hot-search prices for all twelve months.
Protect the logs themselves. Evidence that can be altered is not evidence. Write-once storage, object-lock or immutability settings on the cold tier, strict access controls on the logging platform, and integrity checks all matter when the logs may end up supporting a regulatory finding or a legal case. If your domain administrators can rewrite the security logs, an attacker who owns those accounts can too.
Choosing a model: SaaS SIEM, self-managed, or hybrid
There is no single right answer here, and the honest recommendation depends on your team, your regulatory exposure, and your data volume. The trade-offs, though, are consistent.
DimensionSaaS SIEM (e.g. hosted Splunk, Sentinel)Self-managed open-source stackHybrid (managed hot tier + own cold tier)
Time to value
Fastest; running in days
Slowest; weeks to months of engineering
Moderate
Cost at scale
Rises steeply with volume; hard to cap
Lowest per-GB; you pay in engineering time
Predictable if cold data dominates
Control and visibility
Limited into the vendor's internals
Complete, if you build the telemetry
Good over retained data
Retention economics
Expensive for long windows
Cheap on object storage
Designed around cheap long retention
Operational burden
Low; vendor runs it
High; you run everything
Shared
Lock-in risk
High
Low
Moderate
Best fit
Small security team, moderate volume, fast start
Strong platform team, very high volume, cost pressure
Regulated firm needing long, cheap, auditable retention
For most regulated mid-market firms in the region, the hybrid model wins. Keep a 30 to 90 day hot window in a managed search tier your analysts live in, and push everything to columnar object storage for the full compliance retention period. You get fast investigation on recent data and cheap, immutable, queryable archives for the rest. That is, not coincidentally, close to the shape the digital bank landed on when it rebuilt its own platform and cut its logging cost roughly in half.
Retention: the number that decides your audit and your bill
Retention is where finance, engineering, and compliance argue, and the argument is usually settled by whoever shouts loudest rather than by the actual requirements. Get the requirements on the table first.
Framework or driverRetention expectation for security-relevant logs
PCI DSS v4.0 (Req. 10.5.1)
At least 12 months, with the most recent 3 months immediately available
ISO 27001:2022 (A.8.15)
Defined by risk assessment; commonly 6–12 months for operational logs
NIST CSF 2.0 / NIST 800-92
Risk-based; long enough to support detection and investigation
UAE regulated sectors
Sector rules (e.g. central bank, telecom, critical infrastructure) often mandate longer windows; check your specific obligations
Incident investigation reality
Attacker dwell time is frequently weeks; short windows destroy the trail
The practical move is to set retention per data class, not as one blanket number. Authentication events, privileged-access logs, network flow records, and DNS queries carry high investigative value and deserve long retention. Verbose application debug logs can roll off in days. Tiered storage is what makes this affordable: the high-value, low-volume security logs cost little to keep for a year once they sit in compressed columnar files on object storage.
Detection is only as good as what you keep and parse
Collecting logs is table stakes. Turning them into detection is the work that actually reduces risk, and it depends entirely on the pipeline choices above. Detection rules, whether written in a portable format like Sigma or native to your SIEM, match on fields. If your pipeline has not normalised those fields consistently, every rule becomes brittle and source-specific. If your pipeline dropped the relevant event to save cost, the best rule in the world never fires.
Three things separate a log store from a detection capability. First, coverage: are you actually collecting from the sources that matter, including identity providers, endpoints, cloud control planes, and network gear, not just application servers. Second, parsing and normalisation: can a single rule reason about "a failed privileged login" across every system that can produce one. Third, enrichment: does the analyst see the user's role, the asset's criticality, and any threat-intel context inline, or do they have to pivot through five other tools to assemble it. In our SOC engagements, improving these three usually does more for detection than buying another tool.
This is also where the current wave of AI-assisted security operations lives or dies. Automated triage, alert summarisation, and anomaly detection are only as trustworthy as the underlying data. Feed a model inconsistent, incomplete logs and you get confident nonsense at machine speed. A clean, normalised, well-retained log platform is the unglamorous foundation that makes the newer tooling worth anything.
A log pipeline readiness checklist
Use this as a fast self-assessment. If you cannot answer yes to most of these, the gaps are worth prioritising before your next audit or, less pleasantly, your next incident.
- Ingestion is decoupled from query, so an inbound spike cannot slow investigations.
- A buffer or streaming layer absorbs traffic bursts without dropping events.
- The pipeline emits its own telemetry: ingestion rate, buffer depth, and drop counts are monitored and alerted on.
- Security-relevant events are identified and protected from filtering or sampling, while noise is dropped deliberately.
- Events are normalised to a consistent schema and enriched with identity, asset, and threat context.
- Clocks are synchronised across all log sources, so timelines line up during an investigation.
- Retention is set per data class and meets every applicable framework, with PCI DSS's 12-month floor covered where card data is in scope.
- Stored logs are tamper-resistant: immutable or write-once cold storage, with tight access control on the platform itself.
- Recent data is searchable fast; archived data is still queryable when an auditor or incident demands it.
- Detection content is portable and not locked to a single vendor's proprietary language.
- Coverage spans identity, endpoint, cloud control plane, and network, not just applications.
- Someone owns the platform, reviews its health regularly, and can prove log integrity on request.
How Aydahwa Enterprise Can Help
We build and run this kind of platform for regulated organisations, and we have done it under the constraints that make it hard: fixed budgets, live production systems, and auditors who want evidence rather than intentions. Our work draws on hands-on experience across banking, telecom, and critical national infrastructure, and on certifications including ISO 27001, PCI DSS, SOC 2, and the Microsoft Cybersecurity Architect Expert credential, mapped against NIST CSF and CIS Controls so the outcome stands up to scrutiny.
If your logging costs are climbing without a ceiling, if you are not confident the evidence would be there after an incident, or if a PCI DSS or ISO 27001 assessment is on the horizon, that is precisely the work we do. Our cybersecurity services cover SOC and SIEM design, detection engineering, and log architecture. Our cloud security and migration team handles the object-storage and tiering side on AWS and Azure, and our managed IT support keeps the platform healthy day to day.
Want to gauge where you stand before committing to anything? Start with our free cybersecurity self-assessment and the cybersecurity readiness checklist, then get in touch and we will walk through your specific gaps. Bring your current retention numbers and your last audit findings, and we will tell you plainly what to fix first.
