Skip to main content

Detection & Monitoring Design Patterns

Premium

“How would you design detection for privilege escalation across a large enterprise?”

Detection and monitoring design questions like the one above test whether you can create visibility across systems and translate signals into action. Interviewers want to know if you understand how to collect, correlate, and respond to security telemetry in a way that's scalable, efficient, and minimizes alert fatigue.

Strong candidates demonstrate four core abilities:

  1. Identify key telemetry sources across infrastructure, identity, and application layers
  2. Correlate events to detect real threats with minimal false positives
  3. Design monitoring systems that scale with business and technical growth
  4. Explain tradeoffs between visibility, data volume, and performance

The key is emphasizing signal quality over signal quantity, showing you can detect threats without drowning security teams in noise.

Key telemetry sources

Not all telemetry is created equal. Here are the critical sources and why they matter:

CategoryWhat it revealsExample events to monitor
IdentityWho's authenticating and what privileges they're gainingMFA failures, privilege escalations, new service accounts, unusual login times or locations
NetworkWhere data is flowing and who's communicatingVPC flow logs, DNS queries, unusual outbound connections, data exfiltration patterns
ApplicationHow users interact with your servicesAPI access logs, failed authorization attempts, SQL injection patterns, rate limit violations
EndpointWhat's executing on hostsProcess creation, file modifications, registry changes, malware signatures, persistence mechanisms
Cloud ServicesInfrastructure and configuration changesPublic S3 buckets, IAM policy changes, KMS key usage, security group modifications

Link each telemetry source directly to detectable threats. For example: "I'd monitor CloudTrail for IAM changes because that's how attackers establish persistence after initial compromise. They create new admin users or modify policies to maintain access."

Design & monitoring architecture

The detection pipeline looks like this: Collection → Normalization → Enrichment → Correlation → Alerting → Response

Here's what happens at each stage:

1. Collection

  • Goal: Ingest logs and metrics from multiple sources
  • Sources: CloudTrail, VPC flow logs, application logs, endpoint telemetry, DNS logs
  • Architecture: Stream logs to centralized storage (S3, SIEM) using agents or native integrations

2. Normalization

  • Goal: Parse and standardize events into a common format
  • Why it matters: You can't correlate events if they're in different schemas
  • Example: Transform AWS CloudTrail and GCP Audit Logs into the same schema so you can query both

3. Enrichment

  • Goal: Add context that makes events more actionable
  • Context to add:
    • Asset ownership (who owns this server?)
    • Geolocation (is this login from a known location?)
    • Threat intelligence (is this IP address known malicious?)
    • User context (is this normal behavior for this user?)

4. Correlation

  • Goal: Identify relationships across identity, network, and host activity
  • Why it matters: Single events are often benign; patterns reveal attacks
  • Example: "Failed SSH login" + "Successful login from new IP" + "sudo to root" + "New user created" = likely compromise

5. Alerting

  • Goal: Prioritize actionable alerts; suppress noise
  • Key principle: Alert only when human action is possible or necessary
  • Severity levels: Critical (immediate response), High (same-day), Medium (weekly review)
  • Example: Alert immediately on "admin creating new admin user," suppress "failed login from known bad bot"

6. Response

  • Goal: Trigger automated remediation or human triage
  • Automated response: Revoke compromised access keys, isolate infected hosts, block malicious IPs
  • Human response: Create incident tickets, page on-call engineer, escalate to security team

Detection patterns

Different threats require different detection approaches. Here are the core patterns:

PatternHow it worksBest forExample
Anomaly-BasedEstablish baseline behavior, alert on deviationsDetecting novel attacks or insider threatsAlert on API usage 10x above normal, login from new country
Signature-BasedMatch logs against known attack indicators (IoCs)Detecting known malware, exploits, or threat actorsMatch file hashes against threat intel feeds, detect known SQL injection patterns
Heuristic/Rule-BasedCombine multiple conditions for contextual detectionDetecting attack techniques (MITRE ATT&CK)"Multiple failed logins followed by successful login and immediate token creation"
Behavioral CorrelationLink multiple event types across systems to detect campaignsDetecting sophisticated, multi-stage attacksIAM policy change + unusual S3 access + spike in outbound traffic = data exfiltration

Strong candidates mention multiple detection styles and explain when to use each. For example: "I'd use signature-based detection for known threats where speed matters, anomaly-based for insider threats where behavior matters more than specific indicators, and behavioral correlation for detecting advanced persistent threats that unfold across multiple systems."

Example

Prompt: "You're designing a detection and monitoring system for a large multi-cloud environment. How would you approach it?"

Here's how to structure your response with SALT:

Scope

Define the systems and goals for detection.

"I'm designing detection and monitoring for a multi-cloud environment spanning AWS and GCP that supports:

  • Web applications serving customers
  • API services for mobile and third-party integrations
  • Backend data processing pipelines
  • Internal tools for employees

The detection goals are:

  • Prevent unauthorized access to customer data and infrastructure
  • Detect misconfigurations that expose resources publicly
  • Identify data exfiltration attempts
  • Monitor for privilege escalation and persistence techniques

I'll assume we need to meet compliance requirements like SOC 2 and maintain logs for forensic investigation."

Why this matters: Clear scope prevents you from designing overly broad or insufficiently focused detection systems.

Assets

Identify what you're protecting and what telemetry reveals threats to those assets.

"The critical assets are:

  1. Customer data stored in databases and object storage (S3, Cloud Storage)
  2. Identity and access credentials (IAM keys, service accounts, user sessions)
  3. Infrastructure control plane (ability to modify configurations, deploy code)
  4. Production workloads (uptime and integrity of services)

The key telemetry sources that protect these assets:

  • Identity events: CloudTrail, GCP Audit Logs (authentication, authorization, IAM changes)
  • Network flow logs: VPC flow logs, DNS query logs (data exfiltration, C2 communication)
  • Application logs: API access logs, web server logs (injection attacks, abuse patterns)
  • Cloud service logs: S3 access logs, KMS usage logs (unauthorized data access)
  • Endpoint telemetry: OS logs, process execution (malware, persistence)"

Why this matters: Prioritizing assets helps you focus telemetry collection on what actually matters, not just logging everything.

Layers

We’ll zoom into the detection and monitoring layer to apply the architecture we’ve learned in this lesson.

1. Collection

  • Aggregate logs from multiple sources into a central SIEM:
    • AWS CloudTrail
    • VPC Flow Logs
    • GCP Audit Logs

2. Normalization

  • Convert collected logs into a standard schema (e.g., ECS or JSON) to enable consistent analysis across clouds.

3. Correlation

  • Match high-value actions with user context:
    • Policy changes
    • Key usage
  • Alert on deviations from baseline behavior to detect potential abuse or misconfiguration.

4. Response

  • Automate containment for confirmed incidents (e.g., revoke access keys).
  • Log all actions for compliance and audit purposes.

Tradeoffs

“Centralizing logs from multiple clouds increases storage and processing costs, as all raw and normalized data must be ingested, retained, and indexed in a central SIEM. However, it provides cross-cloud visibility, allowing security teams to detect patterns, correlate events across environments, and respond to incidents more quickly. Without centralization, attacks spanning multiple clouds might go unnoticed, limiting situational awareness.”

Next Lesson