Config Distribution - System Design for Distributing Config Updates at Scale

Question

## Round 1 - System Design

## Problem

Design a configuration distribution system for a large fleet of services. When an operator updates a config key, the change must propagate to all running service instances within 30 seconds. The fleet has ~100,000 service instances across multiple data centers.

Requirements:
- Write path: operator sets `config[key] = value` via an admin API
- Read path: each service reads its config on startup and on change notification
- Consistency: eventual, but stale reads older than 30s are a SLA breach
- Availability: config reads must work even if the central store is temporarily down (use local cache)

## Follow-ups

1. Walk through the data flow: how does a config write reach 100K instances within 30s?
2. How do you handle config versioning and rollback?
3. Compare push (server notifies clients) vs. poll (clients check periodically). Which do you use and why?
4. How do you detect when a service instance is running a stale config version?
5. What happens if a config update causes a service crash -- how do you implement a safe rollout with canary?
6. How does your design change if config values can be large blobs (e.g., 1 MB ML model weights)?

Config Distribution - System Design for Distributing Config Updates at Scale

Question Details

Full Details

Topics

More from Amazon