ML System Design: Design an End-to-End Machine Learning Platform for a Streaming Service

Question

## Round 1 - System Design

## Problem

Design a machine learning platform for a streaming service that trains, evaluates, deploys, and monitors recommendation models at scale. The platform must support multiple teams running concurrent experiments.

**Scope to address:**
- Feature store design and online/offline serving.
- Training pipeline orchestration.
- Model registry and versioning.
- Online serving with SLA guarantees.
- Experiment tracking and A/B testing framework.
- Monitoring for data drift, model drift, and pipeline failures.

## Key Components

**Feature Store**
- Offline: Hive/Spark for batch feature computation.
- Online: Redis for low-latency feature retrieval at inference time.
- Point-in-time correct joins to prevent future leakage in training.

**Training Pipeline**
- Orchestrated via Airflow or Kubeflow Pipelines.
- Triggered on new data arrival or schedule; artifact versioning via MLflow.

**Serving Layer**
- Model registry with staging / production / shadow slots.
- Canary deployments with automatic rollback on metric degradation.

**Monitoring**
- Feature distribution shift (KL divergence alerts).
- Prediction distribution shift.
- Business metric tracking (CTR, watch time) correlated to model versions.

## Follow-ups

1. How do you ensure training-serving skew is minimized in the feature pipeline?
2. What is your strategy for handling cold-start users in the recommendation model?
3. How do you enforce data governance (PII scrubbing) before features reach the training pipeline?

ML System Design: Design an End-to-End Machine Learning Platform for a Streaming Service

Question Details

Full Details

Topics

More from Stripe