SIEM Explained: Security Logs, Correlation & Cost

Security Information and Event Management

A SIEM (security information and event management system) is the place where security-relevant events from across an organization are collected, normalized into a common shape, correlated, searched, and kept. When an alert fires or an incident begins, the SIEM is where responders answer the first questions: who logged in, from where, what changed, and what else that identity touched.

The value of a SIEM depends almost entirely on what goes into it and how it is organized. Ingesting everything produces enormous bills and unsearchable noise; ingesting too little leaves blind spots that are only discovered during an incident. A useful SIEM starts from the questions defenders need to answer and works back to the log sources, fields, and retention that answer them.

TL;DR

Collect the high-value sources first: identity provider, cloud control-plane audit logs, endpoint telemetry, network edge, and application audit logs.
Normalize events into a common schema (OCSF, ECS, or your vendor's model) so one query works across sources.
Enrich events with context — asset owner, user role, geolocation, threat intelligence — at ingest time.
Set retention by need: weeks of fast, searchable data and months or years of cheaper archive for investigations and compliance.
Watch ingestion health: a log source that silently stops is a blind spot an attacker benefits from.
Detections built on the SIEM should be tested and tuned like code — see Security Detection Engineering.

Quick Example

A correlation query over normalized authentication events: many failed logins followed by a success for the same user — a common sign of password guessing that eventually worked.

The query only works because every source writes the same normalized fields — user_name, src_ip, outcome, event_time — regardless of whether the event came from the VPN or a web application.

Core Concepts

What a SIEM Does

High-Value Log Sources

Identity and cloud control-plane logs give the most investigative value per gigabyte; start there.

Normalization Schemas

Different products describe the same event differently. A normalization schema gives them common field names and categories:

OCSF (Open Cybersecurity Schema Framework) — a vendor-neutral, open schema adopted by many security vendors and cloud security data lakes.
ECS (Elastic Common Schema) — Elastic's schema, also used beyond Elastic, and being converged with OpenTelemetry semantic conventions.
Vendor models — for example Splunk's Common Information Model or Microsoft Sentinel's ASIM.

Pick one schema, map sources to it at ingest, and write detections and dashboards against the schema rather than raw vendor fields.

Enrichment

Raw events say "user j.smith logged in from 203.0.113.7". Enriched events add that the user is a finance administrator, the IP geolocates to a country the user has never logged in from, and the IP appears on a threat-intelligence list. Enrichment turns many alerts from "investigate" into "obviously fine" or "obviously bad".

Retention and Cost

SIEM pricing usually scales with data ingested or stored, so volume is a design decision:

Reduce cost by filtering low-value events at the pipeline (debug logs, health checks), sampling noisy high-volume sources, routing bulk data to cheaper storage, and keeping only the fields you use. Pipelines built with tools such as Vector, Fluent Bit, or OpenTelemetry Collector make this routing manageable — see Log Aggregation.

SIEM, SOAR, XDR, and Security Data Lakes

Many organizations combine them: a security data lake for volume and history, a SIEM or XDR for detection and investigation, and automation for repetitive response steps.

Best Practices

Start From Investigation Questions

List the questions responders must answer — who accessed this data, what did this account do in the last 30 days, which hosts contacted this domain — and confirm the SIEM can answer each one. Every missing answer is a log source or field to add.

Synchronize Time and Store UTC

Correlation depends on accurate, comparable timestamps. Keep clocks synchronized with NTP, store timestamps in UTC, and record both the event time and the ingest time to spot delayed sources.

Monitor Ingestion Health

Alert when an important source stops sending data or its volume drops sharply. Attackers disable logging when they can; a silent source should page someone.

Protect the Logs

Restrict who can modify or delete security logs, send copies to storage the logged systems cannot write to, and log access to the SIEM itself. Logs that an attacker can erase are not evidence.

Tie Alerts to Runbooks

Every alert routed to people should link to a short runbook: what it means, how to triage it, and how to escalate. Alerts without runbooks get ignored.

Common Mistakes

Ingesting Everything

Unparsed Raw Logs

Local Timestamps

No Alert for Missing Data

FAQ

What is a SIEM used for?

Collecting security-relevant logs from many systems into one place, normalizing them so they can be queried consistently, detecting suspicious patterns across sources, supporting investigations, and retaining evidence for incident response and audits.

What is the difference between a SIEM and log management?

Log management collects and stores logs, mostly for operations and troubleshooting. A SIEM adds security-specific capabilities: normalization into security schemas, threat-intelligence enrichment, correlation rules and analytics, case management, and long-term evidence retention.

How long should we retain security logs?

Long enough to investigate incidents discovered late — attackers often go unnoticed for weeks or months — and long enough to meet compliance obligations. A common pattern is 30–90 days of fast search and one year or more of cheaper archive, adjusted to regulations and contracts.

Do small teams need a SIEM?

They need the capability, not necessarily a dedicated product. Centralizing identity and cloud audit logs in a log platform with alerting covers the essentials. A dedicated SIEM becomes worthwhile when the number of sources, detections, and compliance requirements grows.

How is a SIEM different from XDR?

XDR products provide tightly integrated detection across a single vendor's security tools, with built-in analytics and response actions. A SIEM is vendor-neutral and collects from anything, at the cost of more integration and tuning work. Many organizations use both.