Why The Unified Security Data Lake Multiplies Everything

When we built the Resolution Intelligence Cloud, one of the early decisions that shaped everything else was committing to a Unified Data Model - a single, consistent schema to which all telemetry, from every source, is normalized at ingestion.

This decision was not without cost. Building normalization logic for every telemetry source we support - and we support many - required significant engineering investment. Maintaining schema consistency as source systems update their event formats is ongoing operational work. The investment was real.

The return on that investment has compounded into every capability we have built since. It fundamentally changes how an enterprise derives value from a modern security data lake. Let me be specific about why.

The Pitfalls of Query-Time Translation

Without a unified data model, cross-source analysis requires translation at query time. A detection rule that correlates authentication events from Windows with process execution events from an endpoint agent with network connection events from a network sensor must carry translation logic for each source system's specific field naming conventions and encoding conventions alongside the actual detection logic. This makes rules complex, brittle, and source-specific. When any source system updates its format, the rules that depend on that source may break. This structural fragility is exactly why traditional dump-and-store security data lake architectures fail to scale operationally.

Raw Log Storage vs. Ingestion Normalization

How Architectural Schema Choice Impacts SOC Performance

With a unified data model, all of these events live in the same schema. "Authentication event" means the same thing whether it came from Windows, Linux, a cloud identity provider, or a SaaS application. "Source IP" is the same field regardless of source. "User identity" has been resolved to a canonical form through the entity resolution layer. Detection logic can focus entirely on the analytical question - what behavioral pattern am I looking for? - without carrying any data quality or translation concerns.

Supercharging Machine Learning and GenAI

The ML models benefit equivalently. A behavioral baseline model trained on UDM-normalized data receives consistent feature vectors regardless of which source systems contributed the data. Models that would otherwise require source-specific versions can be built once and applied everywhere. Training data from multiple customer environments can be combined because the schema is consistent.

LLM-assisted investigation benefits directly. When an analyst asks the LLM to help investigate an alert, the LLM receives context drawn from multiple source systems, all in the same consistent structure. The LLM can reason across sources coherently because the data it is reasoning over is coherent.

Engineering Our Foundation on Chronicle UDM

We chose to build on Google Chronicle's UDM — the Unified Data Model Google had developed for their security data platform — because it provided a comprehensive, well-designed foundation that covered the major event categories and entity types we needed. Our team extended it with the additional classification and enrichment attributes specific to our intelligence requirements.

Six years in, the UDM decision has compounded into every capability we have built. It is the foundational investment that made everything else possible. I would make the same decision again without hesitation.

*Part of my ongoing series on data science and the future of security operations.*

 
About the Author