ML Puzzle Coding - Implement Neural Network Backpropagation Step

Question

## Round 1 - MLE Coding ## Problem Implement the backward pass (backpropagation) for a single fully-connected layer with ReLU activation. You are given the gradient flowing in from the next layer and must compute the gradients with respect to the layer's weights, biases, and input. ```python import numpy as np def relu(x: np.ndarray) -> np.ndarray: ... def relu_grad(x: np.ndarray) -> np.ndarray: ... # 1 where x>0, else 0 def linear_backward( dout: np.ndarray, # gradient from upstream, shape (batch, out_features) x: np.ndarray, # input to this layer, shape (batch, in_features) W: np.ndarray, # weights, shape (in_features, out_features) b: np.ndarray, # biases, shape (out_features,) pre_activation: np.ndarray # Wx + b before ReLU, shape (batch, out_features) ) -> tuple: # Returns (dx, dW, db) ... ``` ``` Shapes check: dout: (32, 128) x: (32, 256) W: (256, 128) pre_activation: (32, 128) dx: (32, 256) <- gradient w.r.t. input dW: (256, 128) <- gradient w.r.t. weights db: (128,) <- gradient w.r.t. bias ``` ## Follow-ups 1. What does multiplying `dout` by `relu_grad(pre_activation)` represent in the chain rule? 2. Why is `dW = x.T @ delta` and not `delta @ x.T`? 3. How would the backward pass change for sigmoid activation instead of ReLU? 4. How would you numerically verify your gradient implementation?

ML Puzzle Coding - Implement Neural Network Backpropagation Step

Question Details

Full Details

Topics

More from OpenAI