Log Sources for Microsoft Sentinel — What Breaks and How to Fix It

1) Title + Hook

Hook:

Treating Microsoft Sentinel like a “Dropbox for logs” is like buying a cargo ship to mail a postcard.
Pouring every signal into your Security Information and Event Management (SIEM) is like turning on every light in a stadium to find your keys—bright, expensive, and still not helpful.

This post shows the anti-patterns that quietly destroy SIEM value—and what to do instead.

2) Why It’s Needed (Context)

Security teams love visibility. Finance teams hate surprise bills. Engineering hates noise.
When log-source design is sloppy, you get: runaway costs, alert fatigue, blind spots, and weak investigations.
Microsoft Sentinel is powerful, but it’s metered. Bad choices at the ingest layer ripple into detect, respond, and retain layers.

3) Core Concepts Explained Simply

A) “Collect-Everything” Ingestion → Huge Costs

Technical definition: Ingesting all available telemetry without scoping by use case, severity, or deduplication—often at high-cost data tables (e.g., SecurityAlert, CommonSecurityLog, Syslog with verbose facilities).
Everyday example: Subscribing to every streaming service “just in case,” then watching YouTube.
Technical example: Forwarding full Endpoint Detection and Response (EDR) raw telemetry and verbose Windows Event Forwarding (WEF) for the same hosts, plus firewall flows at 1:1 cadence—no filters.

B) Logs Collected but Not Used

Technical definition: Sources ingested with no mapped analytics rules, hunting queries, or workbooks.
Everyday example: Paying for a gym you never visit.
Technical example: Shipping detailed DNS logs but no detections/queries reference them; no Kusto Query Language (KQL) saved searches.

C) No Retention & Archival Strategy

Technical definition: Single retention setting for all tables; no hot/cold split, no Azure Data Explorer (ADX) or Azure Blob/Archive offload, and no legal hold mapping.
Everyday example: Keeping all photos on your phone forever—until it’s full right before a trip.
Technical example: 180-day retention for chatty Syslog/CommonSecurityLog tables when only 30 days are needed for detections; no archive to cheaper storage.

D) Custom Logs over Native Connectors

Technical definition: Using custom ingestion (HTTP API, custom tables) instead of Microsoft Sentinel data connectors that provide schemas, Advanced Security Information Model (ASIM) normalization, and content packs.
Everyday example: Cooking from scratch when a healthy, cheaper meal kit exists.
Technical example: Parsing Palo Alto logs via custom functions instead of the native connector and ASIM mapping—losing built-in analytics.

E) Duplicate Telemetry from Multiple Pipelines

Technical definition: Same events reaching Sentinel via parallel paths (e.g., agent + syslog forwarder + third-party pipeline), creating cost bloat and duplicate alerts.
Everyday example: Getting the same bank alerts by SMS, email, app, and phone call—annoying and redundant.
Technical example: Windows events ingested from both Azure Monitor Agent (AMA) and a legacy Log Analytics agent (MMA); cloud audit logs via both native connector and a custom ingestion app.

F) No Log Validation

Technical definition: Lack of pre-ingest checks for schema, timestamps, severity, and required fields; no Service Level Objectives (SLOs) for delay, completeness, or deduplication.
Everyday example: Accepting every delivery without checking the box contents.
Technical example: Timestamps ingested in local time, breaking correlation; device hostname missing → entity mapping fails; uneven daily volume with silent drops.

4) Real-World Case Study

Failure — The $180k Surprise

Situation: A global SaaS firm enabled “everything” from firewalls, proxies, endpoints, and cloud audit logs. No content mapped; no filtering; 180-day retention on all tables.
Impact: Monthly Sentinel bill spiked by 60%. Analysts drowned in duplicate alerts; incident MTTR (Mean Time To Remediate) rose from 9h to 16h.
Lesson: Cost without context adds negative value. Start with use cases → data needed → retention tiering.

Success — Use-Case-Driven Design

Situation: A fintech defined 12 priority detections (credential misuse, exfiltration, MFA bypass). They mapped required fields to ASIM schemas and trimmed sources to those fields.
Impact: 37% ingest reduction, +22% detection precision, 2× faster hunts due to consistent entity mapping.
Lesson: Design sources to serve detections, not the other way around.

5) Action Framework — Prevent → Detect → Respond

Prevent (Design & Cost Control)

Define top 15–20 detections first; list required fields (IP, User, Device, App, Action, Result, Timestamp TZ).
Prefer native connectors + ASIM; only custom when absolutely necessary.
Build ingestion policies: include tables, exclude noise (facility/level filters, sampling for flows).
Implement tiered retention:
- Hot (30–60 days): detection & investigation.
- Cold/Archive (6–12 months+): compliance, rare hunts (use ADX/Blob).
Prevent duplicates: one authoritative pipeline per source; document routing.

Detect (Quality & Coverage)

For each table, create at least one analytic rule and one scheduled query that uses it.
Enforce schema validation in parsing functions; normalize to ASIM.
Track signal health KPIs: daily event count deltas, null critical fields, late arrivals (>10 min), duplication rate.

Respond (Operate & Improve)

Build a workbook: cost by table, events by connector, rule hits by source.
Automate feedback loops: when an analytic fires with low confidence, refine source fields/filters.
Quarterly table review: drop unused sources, move low-value logs to archive, merge pipelines.

6) Key Differences to Keep in Mind

Native vs Custom Ingest — Native brings schemas/content; custom brings flexibility & maintenance.
- Scenario: Choose native for popular firewalls; custom only when niche vendor lacks support.
Hot vs Cold Retention — Hot is for speed; cold is for savings.
- Scenario: Keep 30 days hot for IR (Incident Response); move month 2–12 to archive.
Field Completeness vs Volume — Fewer, richer events beat many shallow events.
- Scenario: Keep DNS with query, response, client IP; drop verbose debug flags.
One Pipeline vs Many — Single route is traceable; multiple routes multiply duplicates.
- Scenario: Consolidate to AMA; retire MMA and third-party forwarders.
Use-Case vs Curiosity — Detections drive data; curiosity drives cost.
- Scenario: Only ingest proxy categories needed for DLP (Data Loss Prevention) alerts.

7) Summary Table

Concept	Definition	Everyday Example	Technical Example
Collect-everything ingestion	Ingest all signals without scoping/filters	Subscribing to every streaming service	EDR + WEF + flow logs all verbose to Sentinel
Unused logs	Data with no rules/queries/workbooks	Paying for a gym you don’t use	DNS ingested but no KQL uses it
No retention strategy	One-size retention; no hot/cold	Keeping all photos on phone forever	180 days on Syslog with no archive
Custom over native	DIY ingestion instead of connectors	Cooking from scratch vs meal kit	Custom Palo Alto parsing vs native + ASIM
Duplicate telemetry	Same events via multiple routes	Bank alerts by SMS/email/app/phone	AMA + MMA + syslog duplicating Windows events
No validation	No checks for schema/time/fields	Accepting packages uninspected	Local-time timestamps; missing hostname

8) ASCII Diagram (Signal Health Funnel)

[Sources] --(validated, deduped)--> [Normalization/ASIM]
             \--x duplicates drop--/          |
                                              v
                                   [Analytic Rules & Hunts]
                                              |
                                              v
                                    [Incidents & Response]
                                              |
                                              v
                           [Retention: Hot 30-60d | Archive 6-12m+]

9) What’s Next

Next in this series: “Designing a Use-Case-First Log Strategy for Sentinel: From Detections to Data Contracts.” We’ll publish a field-tested worksheet to map detections → fields → connectors → retention.

🌞 The Last Sun Rays…

Hook answers:

Sentinel isn’t a dump truck for logs; it’s a tuned sensor grid.
More light (data) isn’t better if it blinds you; focused beams (use-cases) win.

Your move:
What one log source would you drop, filter, or archive tomorrow to improve both signal quality and cost—and what detection would stay intact after that change?

Surya

By profession, a CloudSecurity Consultant; by passion, a storyteller. Through SunExplains, I explain security in simple, relatable terms — connecting technology, trust, and everyday life.

Chapter 2 —How Not to Design Log Sources (with Microsoft Sentinel)