Neural Network Weight Initialization

To get a good learning network we want to start approximating only in the linear parts of the Activation Function (tanh) and then slowly move towards the saturating ends to add nonlinearities into the mix.

$w_i\in \left(-\frac{3}{\sqrt{n}},+\frac{3}{\sqrt{n}}\right)$ where $n$ is the number of weights.

Proof

We want to keep the weights relatively small with Mean around zero and Standard Deviation of one $\tanh \left(\sigma \left({\sum }_{i=1}^n{w}_i{x}_i\right)\stackrel{!}{=}1\right)$ We can easily calculate the Variance and Standard Deviation of the weights with respect to the overall domain $(-r,+r)$. ${\sigma }^2(w)=\frac{1}{2r}{\int }_{-r}^{+r}(w-0{)}^{2}dw=\frac{r^2}{3}\Rightarrow \sigma \left(w_i\right)=\frac{r}{\sqrt{3}}\&\sigma \left(x_i\right)=\frac{1}{\sqrt{3}}$ We then use the Strong Law of Large Numbers to calculate $r$ with $\Rightarrow \sigma \left({\sum }_{i=1}^n{w}_i{x}_i\right)\stackrel{!}{=}\sqrt{n}\sigma \left(w_i{x}_i\right)=\sqrt{n}\frac{r}{3}\approx 1$ and $r=\frac{3}{\sqrt{n}}$ We should thus choose values of $w$ in $(-r,+r)$ to get good learning.

More Ressources

• Initializing neural networks - deeplearning.ai
• Don’t Trust PyTorch to Initialize Your Variables | Aditya Rana Blog