InterviewDB Question · Los Angeles

Big Data Design: Architect a Scalable Pipeline for Petabyte-Scale Log Processing

Onsite
Sign in to bookmark

Question Details

Problem

You are asked to design a system that ingests 10 TB of application logs per day, processes them for anomaly detection and aggregation, and serves dashboards with sub-second query latency. Walk through the architecture.

Requirements:
  - Ingest: 10 TB/day, 100k events/sec peak
  - Processing: real-time aggregation + batch anomaly detection
  - Query: dashboard queries over last 7 days, p99 < 500ms
  - Retention: 90 days hot, 3 years cold
  - Fault tolerance: no data loss

Proposed architecture:

Producers -> Kafka (partitioned by service) -+-> Flink (real-time agg)
                                             |     -> Redis (hot counters)
                                             +-> S3 (raw parquet, partitioned by date)
                                                  -> Spark (batch anomaly, daily)
                                                  -> Trino/Presto (ad-hoc SQL)
                                             Dashboard -> Druid (pre-agg OLAP)

Follow-ups

  1. Why partition Kafka by service rather than by timestamp? What are the tradeoffs?
  2. How do you handle late-arriving events in the Flink streaming layer?
  3. What compaction and partitioning strategy on S3 enables fast Trino queries?
  4. How would you implement exactly-once semantics end-to-end from Kafka to the database?

Full Details

Problem

You are asked to design a system that ingests 10 TB of application logs per day, processes them for anomaly detection and aggregation, and serves dashboards with sub-second query latency. Walk through the architecture.

Requirements:
  - Ingest: 10 TB/day, 100k events/sec peak
  - Processing: real-time aggregation + batch anomaly detection
  - Query: dashboard queries over last 7 days, p99 < 500ms
  - Retention: 90 days hot, 3 years cold
  - Fault tolerance: no data loss

Proposed architecture:

Producers -> Kafka (partitioned by service) -+-> Flink (real-time agg)
                                             |     -> Redis (hot counters)
                                             +-> S3 (raw parquet, partitioned by date)
                                                  -> Spark (batch anomaly, daily)
                                                  -> Trino/Presto (ad-hoc SQL)
                                             Dashboard -> Druid (pre-agg OLAP)

Follow-ups

  1. Why partition Kafka by service rather than by timestamp? What are the tradeoffs?
  2. How do you handle late-arriving events in the Flink streaming layer?
  3. What compaction and partitioning strategy on S3 enables fast Trino queries?
  4. How would you implement exactly-once semantics end-to-end from Kafka to the database?
Free preview. Unlock all Applied Intuition questions →

About This Question

This is a reported interview question from a applied intuition interview during the onsite round.

It covers the following topics: System Design, Sql, Onsite .

Topics

System Design Sql Onsite

More Applied Intuition Interview Questions

InterviewDB Campsite Booking: Find Available Campsites Given Reservation Intervals InterviewDB Job Monitor: Track Long-Running Jobs and Alert on Failures or Timeouts InterviewDB Applied Intuition SWE Phone - Encode String (Strings/Encoding) InterviewDB Applied Intuition SWE Phone - Formula Evaluation (Stack/Parsing) InterviewDB Group Coordinates: Cluster 2D Points by Proximity
← All Applied Intuition questions Applied Intuition Software Engineer Questions → All System Design Questions →