For any binary classification dataset, let \(S_{B} \in \mathbb{R}^{d \times d}\) and \(S_{W} \in \mathbb{R}^{d \times d}\) be the between-class and within-class scatter (covariance) matrices, respectively. The Fisher linear discriminant is defined by \(u^{*} \in \mathbb{R}^{d}\), that maximizes
\[J(u) = \frac{u^{T} S_{B} u}{u^{T} S_{W} u}\]
If \(\lambda = J(u^{*})\), \(S_{W}\) is non-singular and \(S_{B} \neq 0\), then \((u^{*}, \lambda)\) must satisfy which ONE of the following equations?
Note: \(\mathbb{R}\) denotes the set of real numbers.
Answer
(A)
Explanation
At the maximum the gradient of \(J\) is zero. Both matrices are symmetric, so by the quotient rule
\[\nabla J(u) = \frac{2 S_{B} u \,(u^{T} S_{W} u) - 2 S_{W} u \,(u^{T} S_{B} u)}{(u^{T} S_{W} u)^{2}} = 0 \;\Rightarrow\; S_{B} u^{*} = J(u^{*})\, S_{W} u^{*} = \lambda S_{W} u^{*}.\]
Since \(S_{W}\) is non-singular, multiplying by \(S_{W}^{-1}\) gives \(S_{W}^{-1} S_{B} u^{*} = \lambda u^{*}\): \(u^{*}\) is an eigenvector of \(S_{W}^{-1} S_{B}\) for its largest eigenvalue \(\lambda\).
Option B has \(\lambda\) on the wrong side (it would hold with \(1/\lambda\)), C multiplies by \(S_{W}\) instead of its inverse, and D fails because \(J\) does not depend on the length of \(u\), so \(u^{*}\) can be rescaled freely.
Answer: A