The SIEM invoice is how most security teams discover they have a data architecture problem. Ingest-based pricing means every new log source, EDR telemetry, cloud audit trails, identity events, DNS, is a line-item decision, and teams respond the only way the pricing lets them: they filter, sample, and shorten retention. Then an incident investigation needs the exact logs that got dropped, from a month outside the retention window, and the cost of the “savings” becomes visible all at once.
The security data lake grew out of that squeeze. Object storage costs a small fraction of SIEM ingest per terabyte, and the tooling around cheap storage, open table formats, distributed query engines, matured to the point where security teams could realistically keep everything and decide what to do with it later. Whether they should is a more interesting question than the vendor pitches suggest.
What Is a Security Data Lake?
A security data lake is a centralized repository that stores raw security telemetry, logs, events, network metadata, and enrichment data at scale, typically on low-cost object storage, so it can be queried for detection, investigation, threat hunting, and analytics. The defining traits are scale (years of retention become affordable), rawness (data lands in or near its original form rather than being forced through a vendor schema at ingest), and openness (the data belongs to you, in formats multiple tools can query, rather than living inside one product’s proprietary store).
The architectural idea that makes this workable is schema-on-read: instead of normalizing every event at ingest time (schema-on-write, the classic SIEM approach), the lake stores data cheaply and applies structure when a query runs. That inverts the old trade-off. Schema-on-write buys fast, predictable queries at the price of expensive ingest and lossy normalization; schema-on-read buys cheap, complete collection at the price of query-time work and engineering discipline.
A security data lake isn’t a SIEM replacement by default. Most real deployments pair them: high-value, detection-relevant sources flow to the SIEM for real-time correlation, while the full-fidelity stream lands in the lake for hunting, forensics, compliance retention, and the long-lookback questions the SIEM’s window can’t answer.
The Economics and Engineering Behind the Shift
Why SIEM Storage Math Stops Working
Security telemetry volume grows faster than security budgets, cloud adoption alone multiplied the audit surface, and per-GB ingest pricing turns that growth into a forced choice between visibility and cost. The predictable result shows up in breach reviews: the DNS logs that would have caught the exfiltration were never collected, or the intrusion began 400 days ago and retention was 90. Detection engineering suffers too, because analysts write rules against the data they have, not the data they need. The lake’s economic pitch is blunt: stop letting the ingest meter make your visibility decisions.
But the total cost picture is less lopsided than the storage comparison implies. Lakes shift spend from licensing to engineering, someone has to build and maintain pipelines, table formats, partitioning, and access controls, and query compute against poorly organized data gets expensive fast. Teams that treat the lake as “cheap storage, done” usually end up with a swamp: petabytes nobody can query effectively.
OCSF and the Normalization Problem That Won’t Normalize Itself
Raw telemetry from fifty sources means fifty ways of spelling “user logged in,” and schema-on-read doesn’t remove that problem, it just defers it to query time. This is what the Open Cybersecurity Schema Framework (OCSF) exists to fix: an open, vendor-neutral event taxonomy, launched in 2022 by a coalition including AWS and Splunk, that gives security data a common shape without locking it into one product. In practice, teams normalize their highest-value sources into OCSF classes as data lands and leave the long tail raw. The framework’s real value is less philosophical than practical: a detection or an investigation query written once against OCSF fields runs across sources, instead of being rewritten per vendor. Adoption still takes work, mapping a quirky source into OCSF classes is real engineering, and half-mapped data is its own trap.
What AI Investigation Wants From Telemetry
The newest pressure on security data architecture comes from agentic AI. An investigating agent doesn’t read dashboards; it asks questions, hundreds of them per investigation, across identity, endpoint, network, and cloud data. That workload rewards exactly what lakes and open schemas provide: cheap, queryable, broad telemetry with predictable structure. An agent that can check a hypothesis against two years of history produces materially better verdicts than one confined to a 30-day hot window. And the reverse holds, an AI investigation layer sitting on top of sampled, short-retention data inherits every blind spot underneath it.
Consider what a single phishing investigation actually touches: the email gateway logs for the message, identity logs for the click and the login that followed, EDR for what ran on the endpoint, proxy data for the callback domain, and cloud audit trails for anything the session did afterwards. A human analyst assembles that across five consoles. An agent assembles it with queries, and the quality of the verdict depends directly on whether those five data sets are reachable, complete, and old enough to establish what normal looked like before the incident.
Do You Need a Lake, or Do You Need Access?
The Query-in-Place Alternative
Centralizing everything is one answer to the visibility problem. The other is leaving data where it already lives, in the SIEM, the EDR console, the cloud provider’s logging service, the data warehouse, and building a query and semantic layer that can reach all of it on demand. The semantic layer holds the map (what data exists where, what the fields mean, how entities join across systems) and federates queries to the source systems, so an investigation can span all of them without a migration project first. The trade-offs are real in both directions: query-in-place avoids duplicate storage and multi-quarter pipeline builds but depends on source-system query performance and API limits; a lake gives you one fast, uniform store but costs the engineering to fill and maintain it.
The two models also fail differently, which matters for planning. A lake fails slowly and visibly: pipelines lag, a source’s format changes and breaks a parser, storage grows faster than the budget line. Query-in-place fails at query time: the EDR vendor throttles the API during exactly the incident when you need a thousand lookups, or a source system’s search is simply too slow for interactive investigation. Neither failure mode is disqualifying. But you should know which one you’re signing up for before the architecture decision, not during the first major incident after it.
Where an AI SOC Fits in This Choice
This architectural fork matters most when evaluating an AI SOC, because some platforms quietly require you to solve the data problem their way first, re-ingest your telemetry into their store, pay the pipeline tax, then get the AI. The alternative model treats your existing stack as the data layer. Conifers CognitiveSOC™ takes the query-in-place approach through vendor-agnostic AI integrations: its investigation agents query the SIEM, data lake, EDR, and identity systems you already run, where the data already sits, rather than demanding a new central repository. For teams with a lake, the agents use it; for teams without one, the lake stops being a prerequisite. Seeing an investigation fan out across live sources makes the model concrete, that’s demonstrable at a live demo.
When the Lake Is Still the Right Call
None of this makes the lake obsolete. Long compliance retention (PCI DSS, HIPAA, and sector rules that want years of logs), large-scale threat hunting over historical data, detection engineering that needs training and backtesting data, and organizations already staffed with data engineering talent all argue for owning a lake. So does vendor independence: an open-format lake is the one copy of your security history that survives every SIEM migration. The honest framing is that the lake and the semantic layer solve different halves of the problem, one is where truth accumulates, the other is how questions reach it, and mature architectures increasingly have both. Which half you build first depends on whether your acute pain is retention or investigation speed.
Frequently Asked Questions About Security Data Lakes
What’s the difference between a security data lake and a SIEM?
A SIEM is a real-time detection and correlation engine: it normalizes events at ingest, runs rules continuously, and alerts as things happen, with pricing and retention tuned to a hot window of recent data. A security data lake is a storage and analytics substrate: raw telemetry at full fidelity, kept for years, queried on demand for hunting, forensics, and long-lookback investigation. The SIEM answers “what is happening right now that matches known-bad patterns?”; the lake answers “what happened, anywhere, over any period, including things we didn’t have a rule for.” Most enterprises run both, routing detection-relevant streams to the SIEM and everything to the lake, because neither does the other’s job well.
Is a security data lake cheaper than a SIEM?
Per terabyte stored, dramatically. Per usable outcome, it depends. Object storage undercuts SIEM ingest pricing by an order of magnitude or more, which is the entire reason the category exists. But the lake doesn’t come with the SIEM’s built-in parsing, correlation, dashboards, or support; you rebuild those with data engineers, pipeline tooling, and query compute, all of which cost real money and, more scarce, real skilled headcount. Organizations with strong data platform teams routinely come out ahead. Organizations that stand up a lake to escape a licensing bill, without staffing it, tend to buy a cheaper place to lose their data.
When is a security data lake the wrong choice?
When the engineering capacity to run it doesn’t exist and can’t be hired, which describes a lot of mid-size security teams. It’s also the wrong first move when the actual pain is investigation speed rather than retention: if analysts are drowning in a queue today, a multi-quarter pipeline project doesn’t help this quarter, and a query-in-place investigation layer over existing tools attacks the real bottleneck sooner. And small environments with modest telemetry volumes may find modern SIEM tiers or their cloud provider’s native logging perfectly adequate, the lake’s economics only dominate at scale. The pattern breaks down, in short, wherever storage was never the binding constraint.
Do AI SOC platforms require a security data lake?
No, and it’s a useful evaluation question precisely because platforms answer it differently. Some are architected around their own centralized store and effectively make re-ingestion the price of admission; others, including AI SOC platforms built on federated, vendor-agnostic integration, query your existing SIEM, lake, EDR, and identity systems in place. If you already operate a lake, either model can use it. If you don’t, the federated model means AI-driven investigation isn’t gated behind a data re-platforming project. Ask vendors directly: what data must move into your store, who pays for that pipeline, and what happens to our history if we leave?