Linear Discriminant Analysis

A Supervised Learning type of Dimensionality Reduction. It takes class labels into account.

It tries to find the feature subspace that optimizes class separability.

Assumptions

• normally distributed
• identical Covariance matrices
• independent

but still works reasonably well with slight violations of these assumptions.

General Idea

Steps

1. Standardization
2. d-dimensional mean vector for every class
3. Between-Class Scatter Matrix and Within-Class Scatter Matrix
4. Eigenvectors and Eigenvalues for $S_W^{-1}{S}_B$
5. Sort
6. Choose top k and create transformation matrix
7. Project into subspace with transformation matrix