Consider the two neural networks (NNs) shown in Figures 1 and 2, with \(ReLU\) activation (\(ReLU(z) = \max\{0, z\}, \forall z \in \mathbb{R}\)). \(\mathbb{R}\) denotes the set of real numbers. The connections and their corresponding weights are shown in the Figures. The biases at every neuron are set to 0. For what values of \(p, q, r\) in Figure 2 are the two NNs equivalent, when \(x_{1}, x_{2}, x_{3}\) are positive?
Figure 1
Figure 2
Answer
(A)
Explanation
The inputs and all weights are positive, so every neuron receives a positive value and \(ReLU\) acts as the identity. Both networks are then linear, and only the total weight from each input to the output matters.
- First layer (weights 1): the neurons receive \(x_{1} + x_{2}\), \(x_{1} + x_{3}\) and \(x_{1} + x_{2} + x_{3}\), which add up to \(s = 3x_{1} + 2x_{2} + 2x_{3}\).
- Second layer: each of its two neurons receives \(2s\).
- Output: \(3(2s) + 3(2s) = 12s\).
So Figure 1 computes \(36x_{1} + 24x_{2} + 24x_{3}\), giving \(p = 36\), \(q = 24\), \(r = 24\). Option D would need \(x_{2}\) and \(x_{3}\) to reach all three first-layer neurons, but each has only two connections.
Answer: A