Mini-Batch Gradient Descent

A widely used Gradient Descent algorithm that is SGD on batches. Instead of updating weights for every individual sample we create batches and accumulate all Gradients in a batch before doing the weight update.

This will result in some more speedup and smooths some of the stochasticity that is introduced by SGD.

The update rule is $\Delta w=-\eta \left(\frac{1}{\text{ batch }}{\sum }_{t=1}^{\text{batch }}{g}_t\right)$where $1\le \text{batch}\le \text{\#data}$ is the batch size.