Performance Modeling: Predict System Latency Under Variable Load Using ML

Question

## Problem

You are given historical telemetry from a distributed service: `(timestamp, qps, p50_latency_ms, p99_latency_ms, error_rate, cpu_util)`. Build a model to predict `p99_latency_ms` and `error_rate` given a future `qps` and `cpu_util`. Walk through:

1. **Feature engineering** — what features to derive from raw telemetry
2. **Model selection** — linear regression, gradient boosting, or neural network; justify your choice
3. **Evaluation** — what metrics matter for ops use cases (MAPE, RMSE, quantile loss?)
4. **Serving** — how the model is used in a capacity planning workflow

## Example Scenario

```
Historical data shows:
  qps=500  -> p99=20ms,  error_rate=0.1%
  qps=800  -> p99=45ms,  error_rate=0.5%
  qps=1000 -> p99=200ms, error_rate=5%   # near saturation

Question: predict p99 and error_rate at qps=900 given cpu_util=75%
```

## Follow-ups

1. How do you handle concept drift when the underlying system changes (e.g., new hardware)?
2. The latency vs. QPS relationship is nonlinear near saturation — how does your model capture that?
3. How would you quantify uncertainty in your predictions for risk-aware capacity planning?

Performance Modeling: Predict System Latency Under Variable Load Using ML

Question Details

Full Details

Topics

More from Anthropic