ML Integration: Design the Integration Layer Between an ML Model and a Production API

Question

## Round 1 - System Design

## Problem

You have trained a recommendation model (collaborative filtering, ~500ms inference time). Design the integration layer that serves this model as part of a production API handling 10,000 requests per second.

**Constraints:**
- p99 latency target: 200ms end-to-end.
- Model is updated daily with a full retrain.
- Fallback required if the model is unavailable.

## Key Design Decisions

**Serving Infrastructure**
- Model server options: TorchServe, Triton, custom FastAPI. Trade-offs?
- How do you handle the 500ms inference time given a 200ms latency budget?

**Caching**
- Pre-compute recommendations for top 10% most active users.
- Cache invalidation on model update.
- Cache key design: user_id + context hash (device, time-of-day bucket).

**Fallback Strategy**
- Popularity-based fallback when model is unreachable.
- Circuit breaker pattern to avoid cascading failures.

**Model Rollout**
- Shadow mode: new model runs alongside old, compare outputs before full cutover.
- Canary: route 5% of traffic to new model, monitor click-through rate before promoting.

## Follow-ups

1. How do you detect model degradation in production without labeled ground truth in real time?
2. What monitoring signals alert you to a model update causing a regression?
3. How would you A/B test two models while controlling for novelty effects?

ML Integration: Design the Integration Layer Between an ML Model and a Production API

Question Details

Full Details

Topics

More from Stripe