1) Title + Hook
Hook:
- Treating Microsoft Sentinel like a “Dropbox for logs” is like buying a cargo ship to mail a postcard.
- Pouring every signal into your Security Information and Event Management (SIEM) is like turning on every light in a stadium to find your keys—bright, expensive, and still not helpful.
This post shows the anti-patterns that quietly destroy SIEM value—and what to do instead.
2) Why It’s Needed (Context)
Security teams love visibility. Finance teams hate surprise bills. Engineering hates noise.
When log-source design is sloppy, you get: runaway costs, alert fatigue, blind spots, and weak investigations.
Microsoft Sentinel is powerful, but it’s metered. Bad choices at the ingest layer ripple into detect, respond, and retain layers.
3) Core Concepts Explained Simply
A) “Collect-Everything” Ingestion → Huge Costs
- Technical definition: Ingesting all available telemetry without scoping by use case, severity, or deduplication—often at high-cost data tables (e.g.,
SecurityAlert,CommonSecurityLog,Syslogwith verbose facilities). - Everyday example: Subscribing to every streaming service “just in case,” then watching YouTube.
- Technical example: Forwarding full Endpoint Detection and Response (EDR) raw telemetry and verbose Windows Event Forwarding (WEF) for the same hosts, plus firewall flows at 1:1 cadence—no filters.
B) Logs Collected but Not Used
- Technical definition: Sources ingested with no mapped analytics rules, hunting queries, or workbooks.
- Everyday example: Paying for a gym you never visit.
- Technical example: Shipping detailed DNS logs but no detections/queries reference them; no Kusto Query Language (KQL) saved searches.
C) No Retention & Archival Strategy
- Technical definition: Single retention setting for all tables; no hot/cold split, no Azure Data Explorer (ADX) or Azure Blob/Archive offload, and no legal hold mapping.
- Everyday example: Keeping all photos on your phone forever—until it’s full right before a trip.
- Technical example: 180-day retention for chatty
Syslog/CommonSecurityLogtables when only 30 days are needed for detections; no archive to cheaper storage.
D) Custom Logs over Native Connectors
- Technical definition: Using custom ingestion (HTTP API, custom tables) instead of Microsoft Sentinel data connectors that provide schemas, Advanced Security Information Model (ASIM) normalization, and content packs.
- Everyday example: Cooking from scratch when a healthy, cheaper meal kit exists.
- Technical example: Parsing Palo Alto logs via custom functions instead of the native connector and ASIM mapping—losing built-in analytics.
E) Duplicate Telemetry from Multiple Pipelines
- Technical definition: Same events reaching Sentinel via parallel paths (e.g., agent + syslog forwarder + third-party pipeline), creating cost bloat and duplicate alerts.
- Everyday example: Getting the same bank alerts by SMS, email, app, and phone call—annoying and redundant.
- Technical example: Windows events ingested from both Azure Monitor Agent (AMA) and a legacy Log Analytics agent (MMA); cloud audit logs via both native connector and a custom ingestion app.
F) No Log Validation
- Technical definition: Lack of pre-ingest checks for schema, timestamps, severity, and required fields; no Service Level Objectives (SLOs) for delay, completeness, or deduplication.
- Everyday example: Accepting every delivery without checking the box contents.
- Technical example: Timestamps ingested in local time, breaking correlation; device hostname missing → entity mapping fails; uneven daily volume with silent drops.
4) Real-World Case Study
Failure — The $180k Surprise
- Situation: A global SaaS firm enabled “everything” from firewalls, proxies, endpoints, and cloud audit logs. No content mapped; no filtering; 180-day retention on all tables.
- Impact: Monthly Sentinel bill spiked by 60%. Analysts drowned in duplicate alerts; incident MTTR (Mean Time To Remediate) rose from 9h to 16h.
- Lesson: Cost without context adds negative value. Start with use cases → data needed → retention tiering.
Success — Use-Case-Driven Design
- Situation: A fintech defined 12 priority detections (credential misuse, exfiltration, MFA bypass). They mapped required fields to ASIM schemas and trimmed sources to those fields.
- Impact: 37% ingest reduction, +22% detection precision, 2× faster hunts due to consistent entity mapping.
- Lesson: Design sources to serve detections, not the other way around.
5) Action Framework — Prevent → Detect → Respond
Prevent (Design & Cost Control)
- Define top 15–20 detections first; list required fields (IP, User, Device, App, Action, Result, Timestamp TZ).
- Prefer native connectors + ASIM; only custom when absolutely necessary.
- Build ingestion policies: include tables, exclude noise (facility/level filters, sampling for flows).
- Implement tiered retention:
- Hot (30–60 days): detection & investigation.
- Cold/Archive (6–12 months+): compliance, rare hunts (use ADX/Blob).
- Prevent duplicates: one authoritative pipeline per source; document routing.
Detect (Quality & Coverage)
- For each table, create at least one analytic rule and one scheduled query that uses it.
- Enforce schema validation in parsing functions; normalize to ASIM.
- Track signal health KPIs: daily event count deltas, null critical fields, late arrivals (>10 min), duplication rate.
Respond (Operate & Improve)
- Build a workbook: cost by table, events by connector, rule hits by source.
- Automate feedback loops: when an analytic fires with low confidence, refine source fields/filters.
- Quarterly table review: drop unused sources, move low-value logs to archive, merge pipelines.
6) Key Differences to Keep in Mind
- Native vs Custom Ingest — Native brings schemas/content; custom brings flexibility & maintenance.
- Scenario: Choose native for popular firewalls; custom only when niche vendor lacks support.
- Hot vs Cold Retention — Hot is for speed; cold is for savings.
- Scenario: Keep 30 days hot for IR (Incident Response); move month 2–12 to archive.
- Field Completeness vs Volume — Fewer, richer events beat many shallow events.
- Scenario: Keep DNS with query, response, client IP; drop verbose debug flags.
- One Pipeline vs Many — Single route is traceable; multiple routes multiply duplicates.
- Scenario: Consolidate to AMA; retire MMA and third-party forwarders.
- Use-Case vs Curiosity — Detections drive data; curiosity drives cost.
- Scenario: Only ingest proxy categories needed for DLP (Data Loss Prevention) alerts.
7) Summary Table
| Concept | Definition | Everyday Example | Technical Example |
|---|---|---|---|
| Collect-everything ingestion | Ingest all signals without scoping/filters | Subscribing to every streaming service | EDR + WEF + flow logs all verbose to Sentinel |
| Unused logs | Data with no rules/queries/workbooks | Paying for a gym you don’t use | DNS ingested but no KQL uses it |
| No retention strategy | One-size retention; no hot/cold | Keeping all photos on phone forever | 180 days on Syslog with no archive |
| Custom over native | DIY ingestion instead of connectors | Cooking from scratch vs meal kit | Custom Palo Alto parsing vs native + ASIM |
| Duplicate telemetry | Same events via multiple routes | Bank alerts by SMS/email/app/phone | AMA + MMA + syslog duplicating Windows events |
| No validation | No checks for schema/time/fields | Accepting packages uninspected | Local-time timestamps; missing hostname |
8) ASCII Diagram (Signal Health Funnel)
[Sources] --(validated, deduped)--> [Normalization/ASIM]
\--x duplicates drop--/ |
v
[Analytic Rules & Hunts]
|
v
[Incidents & Response]
|
v
[Retention: Hot 30-60d | Archive 6-12m+]
9) What’s Next
Next in this series: “Designing a Use-Case-First Log Strategy for Sentinel: From Detections to Data Contracts.” We’ll publish a field-tested worksheet to map detections → fields → connectors → retention.
🌞 The Last Sun Rays…
Hook answers:
- Sentinel isn’t a dump truck for logs; it’s a tuned sensor grid.
- More light (data) isn’t better if it blinds you; focused beams (use-cases) win.
Your move:
What one log source would you drop, filter, or archive tomorrow to improve both signal quality and cost—and what detection would stay intact after that change?
By profession, a CloudSecurity Consultant; by passion, a storyteller. Through SunExplains, I explain security in simple, relatable terms — connecting technology, trust, and everyday life.