Regularisation

Having a big Hypothesis Space can lead to Overfitting. To reduce this overfitting we can try to penalize a hypothesis that is too complex by adding its complexity to the Empirical Loss and therefore increasing the total cost of it.

Minimizing this regularized version of the Empirical Loss

is called regularization. We can use the parameter $\lambda$ to control how much influence the regularization should have.

Minimum Description Length

Methods:

• Lasso Regression
• Ridge Regression
• Elastic-Net Regression
• Dropout
• Early Stopping
• Square Weight Decay