Uber | System Design Round | L5

Recently went through a system design round at Uber where the prompt was: "Design a distributed message broker similar to Apache Kafka." The requirements focused on topic-based pub/sub, partitioned or

Recently went through a system design round at Uber where the prompt was: "Design a distributed message broker similar to Apache Kafka." The requirements focused on topic-based pub/sub, partitioned ordered storage, durability, consumer groups with parallel consumption, and at-least-once delivery. I thought the discussion went really well—covered a ton of depth, including real Kafka internals and evolutions—but ended up with some frustrating feedback. 1. Requirements Clarification Functional: Topics, publish/subscribe, ordered messages per partition, consumer groups for parallel processing, at-least-once guarantees via consumer acks. Non-functional: High throughput/low latency, durability (persistence to disk), scalability, fault tolerance. Probed on push vs. pull model → settled on pull-based (consumer polls). 2. High-Level Architecture Core Components: Brokers clustered for scalability. Topics → Partitions → Replicas (primary + secondaries for fault tolerance). Producers publish to topics (key-based partitioning for ordering). Consumers in groups, with one-to-many consumer-to-partition mapping for parallelism. Coordination: Initially Zookeeper based node manager for metadata, leader election, and consumer offsets—but explicitly discussed evolution to KRaft (quorum-based controller, no external dependency) as a more modern direction. Frontend Layer: Introduced a lightweight proxy layer for dumb clients. Smart clients bypass it and talk directly to brokers after fetching metadata. 3. Deep Dives & Trade-offs This is where I went deep: Storage & Durability: Write-ahead log style: Messages appended to partition segments on disk. Page cache leverage for fast reads. In-sync replicas (ISR) concept: Leader waits for ack from ISR before committing. Replication & Failure Handling: Primary host per partition, secondaries for redundancy. Mix of sync (for durability) and async (for latency) replication. Leader election via ZAB (Zookeeper Atomic Broadcast) for strong consistency and quorum handling during network partitions or broker failures. Producer Side: Serialized operations at partition level for ordering. Key-based partitioning. Consumer Side: Poll + explicit ack for at-least-once guarantees. Offset tracking per consumer group/partition. Parallel consumption within groups. Rebalancing & Assignment: Partition assignment: Round-robin or resource-aware, ensuring replicas not co-located. Coordination: Used a flag (e.g., in Redis or metadata store) to pause consumers during rebalance. Discussed that this can evolve toward Zookeeper based rebalancing in mature systems. Scalability Topics: Adding/removing brokers: Reassign partitions via controller. In sync replicas to ensure higher partition level scalability. 4. Other Advanced Points Explicitly highlighted Kafka's real evolution: From heavy Zookeeper dependency → KRaft for self-managed quorum. Trade-offs such as durability vs. latency (sync acks). Overall, I felt that the interview went quite well and was expecting Hire at least from the round. Considering other rounds were also postivie only I felt that I had more than 50% chance of being selected. However, to my horror I was told that I might only be eligible for L4 as there were callouts in relation to not asking enough calrifying questions. Since LLD, DSA and Managerial rounds went well and this problem itself was not very vague I can't seem to figure out what went wrong. My guess is that there are too many candidates so they end up finding weird reasons to reject candidates. To top it all, they rescheduled my interviews like 5-6 times and I had to keep on brushing up my concepts https://preview.redd.it/09d8bbuzm9hg1.png?width=1770&format=png&auto=webp&s=8a0ea058ad5edb1099f7a7abde7247f58c5adf9b