An effective SIEM is not a product you plug into the wall — it is an engineering pipeline that turns billions of raw events into high-fidelity alerts. This reference covers the internal architecture, the design of correlation rules and the tuning techniques that separate an operational SIEM from an expensive log repository.
Architecture of a modern SIEM
The architecture of a SIEM breaks down into six functional layers. Understanding each one is a prerequisite for optimizing performance, controlling cost and increasing detection capability.
1. Collection and forwarding (Forwarders)
Ingestion starts at the forwarders — agents installed on endpoints and servers, or connectors that receive syslog, Kafka, API pull or file beat. The local forwarder reduces latency and enables edge filtering before consuming bandwidth. In cloud environments, the equivalent is native integrations: AWS CloudTrail delivered via S3 or Kinesis, Azure Monitor via Event Hub, GCP via Pub/Sub. The golden rule is to never send raw data without at least one basic filtering step — heartbeats, application debug logs and low-value system events should be discarded before entering the ingestion queue.
2. Normalization and parsing
Logs arrive in dozens of formats: CEF, LEEF, JSON, syslog RFC 3164, syslog RFC 5424, free text. The parser extracts structured fields and maps them to a common schema. The most widely adopted are the Elastic Common Schema (ECS) for Elasticsearch-based stacks and Splunk's Common Information Model (CIM). Mandatory fields in any schema: timestamp in UTC with millisecond precision, source.ip, destination.ip, user.name, host.name, event.action and event.outcome. Correct normalization is what allows a single rule to detect brute force regardless of whether the event came from an Active Directory, a VPN gateway or a Linux system with PAM.
3. Indexing and storage
The index turns events into searchable structures. Modern architectures separate hot storage (NVMe disks, 30 to 90 days of retention, real-time queries) from warm and cold storage (object storage such as S3 or Azure Blob, 1 to 7 years of retention, cost per GB lower by one or two orders of magnitude). Correct sizing requires knowing the average and peak EPS of each source, the average event size after parsing and the compression ratio — on average, text logs compress 4:1 to 8:1 with algorithms such as Zstandard. An organization with 5,000 users typically generates between 3,000 and 15,000 EPS depending on source coverage.
4. Correlation engine
The correlation engine is the core of the SIEM: it evaluates events against rules and generates alerts when conditions are met. There are two main paradigms. The first is based on sequence rules — the engine maintains a temporal state and fires when a specific sequence of events occurs within a time window, for example, an authentication failure followed by a success followed by the creation of a privileged account in less than ten minutes on the same host. The second is based on statistics and behavior — the engine computes baselines per user, host or entity and alerts when the deviation exceeds a threshold, covering insider threats and compromised accounts whose behavior is anomalous but does not match any known signature.
5. Alerting and prioritization
Alerts generated by the correlation engine need a prioritization system before reaching the analyst's queue. Effective prioritization combines three dimensions: rule severity (critical, high, medium, low), criticality of the affected asset (production server versus development workstation) and user context (privileged account versus regular user). The result is a risk score that orders the queue and allows critical alerts on critical assets from privileged accounts to appear at the top, regardless of when they were generated.
6. Dashboards and reports
Operational dashboards for the tier-1 analyst show the alert queue, triage SLA, MTTD and MTTR in real time. Tactical dashboards for the SOC manager display MITRE ATT&CK coverage, ingestion volume per source, FP rate per rule and weekly trends. Executive reports present confirmed incidents, containment time and avoided impact. Separating these three audiences prevents the analyst's panel from turning into a wall of charts irrelevant to immediate triage.
Log sources and detection value
Not every log is worth the same. The table below summarizes the main sources by the detection capability they offer:
| Source | MITRE techniques covered | Typical volume | Ingestion priority |
|---|---|---|---|
| Active Directory / LDAP (events 4624, 4625, 4768, 4769, 4771) | T1078, T1110, T1558, T1021 | Medium | P0 — mandatory |
| External DNS | T1071.004, T1568, T1048 | High | P0 — mandatory |
| EDR (process and network telemetry) | T1059, T1055, T1003, T1057 | Very high | P0 — mandatory |
| Firewall / NGF | T1046, T1571, T1090 | High | P1 — high priority |
| Web proxy / gateway | T1071.001, T1567, T1105 | Very high | P1 — high priority |
| Cloud trail (AWS/Azure/GCP) | T1078.004, T1530, T1537 | Medium-high | P1 — high priority |
| VPN / remote access | T1133, T1078 | Low-medium | P1 — high priority |
| Email (gateway, headers) | T1566, T1598, T1534 | Medium | P2 |
| Database (audit log) | T1005, T1213, T1190 | Variable | P2 — if sensitive data |
| Linux server syslog (auditd) | T1548, T1053, T1136 | Medium | P2 |
| IoT / OT / SCADA | T0840, T0856 | Variable | P3 — as scoped |
Correlation rules: design and use cases
Signature rules
Signature rules are the simplest and fastest to implement. Concrete examples of high-value use cases: detection of LOLBins (native Windows binaries abused, such as mshta.exe executing an external URL, certutil.exe with the -decode flag or wmic.exe spawning a child process); detection of Kerberoasting (4769 events with RC4-HMAC cipher type from service accounts); and DNS exfiltration (anomalous volume of TXT queries or long subdomains to a specific external domain). Each of these rules has a reference implementation in the SigmaHQ/sigma repository on GitHub, tested against dozens of SIEMs.
Sigma format and detection-as-code
A Sigma rule has the following canonical structure: metadata fields (title, id, status, description, author, date, tags with MITRE ATT&CK references), the logsource section that defines category and product, the detection section with the selectors and logical conditions, and the falsepositives, level and fields fields. Conversion to the target platform — for example, Splunk SPL, Microsoft Sentinel KQL or Elasticsearch Query DSL — is handled by the pySigma tool. The recommended detection-as-code workflow is: write the rule in Sigma, open a pull request in the detection repository, run automated tests with synthetic event datasets, approve and merge, and trigger a CI/CD pipeline that converts and deploys the rule to the production SIEM. This workflow ensures traceability, peer review and immediate rollback capability.
Behavioral rules and UEBA
Behavioral rules require 14 to 30 days of learning to build per-entity baselines. Typical use cases: access outside the user's usual hours, download volume above 3 standard deviations, impossible travel (authentication from two countries in an impossible interval), and a privilege-escalation sequence in a short window. Their weakness is sensitivity to context — migrations and new projects distort the baseline. The antidote is contextual enrichment: if the user is on a migration project registered in the ITSM, the alert for access to new servers should have its severity reduced automatically.
Systematic false-positive tuning
Tuning is not a one-off event — it is a continuous process with weekly and monthly cycles. The four-layer methodology ensures consistent FP reduction without eliminating true positives:
Source suppression: before applying any rule, filter events generated by known, low-risk sources — internal vulnerability scanners (Nessus, Qualys, Tenable), monitoring agents (Zabbix, Datadog), backup service accounts and cross-system integration accounts. Keep this list in a CMDB or versioned spreadsheet and review it quarterly.
Threshold adjustment: static thresholds generate FP in dynamic environments. Use dynamic thresholds based on percentiles — for example, firing when the event exceeds the 99th percentile of the last 30 days of history for that entity, rather than a fixed absolute value.
Context enrichment: add attributes to the event that allow modulating severity: asset criticality (P1 to P4), account type (human or service), presence of compensating controls (MFA active, EDR installed) and the rule's FP history on that host. A rule that fires FP 60% of the time on a specific host should have its severity reduced automatically for that host until the adjustment is made.
Structured feedback loop: deploy a triage system that records the analyst's verdict on each alert. Weekly, calculate the FP rate per rule. Any rule with FP above 30% enters the tuning backlog with high priority. Rules with 100% FP for more than two weeks should be disabled or completely rewritten.
Ingestion pipeline and cost control (EPS and GB)
On platforms with volume-based licensing — Splunk, Microsoft Sentinel, Elastic Security — cost grows with ingested volume. Three reduction techniques with low impact on coverage: (1) forwarder-side filtering of audit success events for read operations on non-critical systems; (2) sampling of internal NetFlow retaining 10% of flows and prioritizing external destinations; (3) aggregation of repeated events — the same firewall deny 10,000 times per minute for the same pair becomes a single event with a counter. Never apply filtering or sampling to authentication, EDR, external DNS or cloud trail — the cost of a missed detection outweighs that of storage.
Integration with SOAR and EDR
The SIEM-SOAR-EDR integration forms the automated response triangle. The SIEM detects and enriches; the SOAR orchestrates actions; the EDR executes containment on the endpoint. The integration flow for an early-stage ransomware case illustrates the potential well: the EDR detects vssadmin delete shadows (T1490), sends the event to the SIEM, which correlates it with earlier events of connection to a suspicious IP and execution of an encoded PowerShell script, generates a critical alert and forwards it to the SOAR. The SOAR runs the playbook: isolates the host via the EDR API, collects the process memory via forensic dump, blocks the C2 IP on the firewall, temporarily suspends the user's account in AD, opens a P1 ticket in the ITSM and notifies the on-call analyst with a timeline summary. All of this happens in under 90 seconds, before the ransomware can spread laterally.
SIEM maturity metrics
Five metrics summarize the health of detection engineering: MTTD — target below 15 minutes for active attacks; MTTR — target below 4 hours for critical incidents; FP rate — target below 20% in aggregate; ATT&CK coverage — percentage of techniques relevant to the sector with at least one active rule, target above 60%; and ingestion growth per source — growth above 30% week over week warrants immediate investigation, as it may indicate a misconfiguration or the start of exfiltration via log amplification.
References
- SigmaHQ/sigma — detection rules in open format (github.com/SigmaHQ/sigma)
- MITRE ATT&CK — adversarial tactics, techniques and procedures (attack.mitre.org)
- NIST SP 800-92 — Guide to Computer Security Log Management
- Elastic Common Schema (ECS) and OCSF — open normalization schemas for security logs
Frequently asked questions
What is the difference between a signature rule and a behavioral rule in a SIEM?
Signature rules compare events to fixed patterns — for example, a known malware hash or a specific string in a firewall log. They are deterministic and generate few false positives when well maintained, but fail against novel threats. Behavioral rules detect statistical deviations: a user who has never accessed the backup server suddenly does so at 3 a.m. Combining the two approaches is what enables broad coverage with adequate precision.
What is Sigma and why does it matter for detection engineering?
Sigma is an open detection-rule format for SIEMs, maintained by the SigmaHQ community on GitHub. A Sigma rule describes the detection logic in a platform-agnostic way and is converted into native syntax by tools such as pySigma. The practical benefit is detection-as-code: rules versioned in a Git repository, reviewed via pull request, tested automatically and deployed in a traceable way. MITRE ATT&CK references the techniques Sigma covers, making it the standard detection vocabulary between intelligence teams and the SOC.
How do I calculate a SIEM's ingestion cost and avoid EPS explosion?
Ingestion cost depends on events per second (EPS) and, on data-volume-based platforms, on GB/day. To reduce volume without losing coverage, filter very-low-value events at the edge — monitoring heartbeats, known internal scans. Use sampling only for high-volume, low-risk telemetry, such as NetFlow from internal links. Internal DNS logging generates enormous volume but is indispensable for C2 detection; the solution is to retain only external queries and previously unseen domains.
Which log sources have the highest detection value and should enter the SIEM first?
The most common sequence starts with: (1) authentication logs — Active Directory/LDAP, VPN, SSO; (2) external DNS and web proxy/gateway; (3) EDR; (4) firewall and NDR; (5) cloud trail (AWS CloudTrail, Azure Activity Log, GCP Audit Logs); and (6) critical application logs. Sources such as printers and generic IoT have marginal value and should enter only after the coverage baseline is consolidated.
What is SIEM tuning and how do I reduce false positives systematically?
Tuning is the continuous process of adjusting rules, thresholds and allowlists so the SIEM produces high-fidelity alerts. The approach uses four layers: source suppression, threshold adjustment, context enrichment and feedback loop. The goal is not zero alerts, but a high proportion of actionable alerts — below 20% FP in aggregate is the maturity benchmark.
How do I integrate a SIEM with SOAR and EDR for automated response?
The integration operates in three layers: the SIEM enriches the alert and forwards it to the SOAR via API; the SOAR runs playbooks — isolation via EDR, blocking at the firewall, account suspension in AD, opening a ticket in the ITSM; the analyst receives the already-triaged alert with collected evidence and actions taken. Destructive actions that affect production services should always require human confirmation.
Decripte operates a 24x7 SOC with dedicated detection engineering, integrating SIEM, SOAR and EDR for organizations from 1 to more than 100,000 employees. The Threat Management plan includes event correlation, continuous tuning and traceable MITRE ATT&CK coverage — available free for the first diagnostic. Explore the plans or start free.