Which of the following statements is/are correct about the rectified linear unit (ReLU) activation function defined as \(\mathrm{ReLU}(x) = \max(x, 0)\), where \(x \in \mathbb{R}\)?
Answer
(A), (C)
Explanation
ReLU is 0 for \(x \lt 0\) and equals \(x\) for \(x \ge 0\).
- A (correct): both pieces are continuous and meet at the value 0 when \(x = 0\), so there is no jump.
- B (wrong), C (correct): at \(x = 0\) the left derivative is 0 and the right derivative is 1. They differ, so the graph has a corner there and ReLU is not differentiable at 0. (Training code simply uses 0 or 1 as the gradient at that point.)
- D (wrong): with \(x = 1\) and \(a = 2\), \(\mathrm{ReLU}(1) = 1\) but \(\mathrm{ReLU}(2) = 2\). What does hold is \(\mathrm{ReLU}(ax) = a\,\mathrm{ReLU}(x)\) for \(a \ge 0\).
Answer: A, C