Neural Network Weight Initialization

To get a good learning network we want to start approximating only in the linear parts of the Activation Function (tanh) and then slowly move towards the saturating ends to add nonlinearities into the mix.

wi∈(−3n,+3n)w_i \in\left(-\frac{3}{\sqrt{n}},+\frac{3}{\sqrt{n}}\right)

where nn is the number of weights.

Proof

We want to keep the weights relatively small with Mean around zero and Standard Deviation of one

tanh⁡(σ(∑i=1nwixi)=!1)\tanh \left(\sigma\left(\sum_{i=1}^n w_i x_i\right) \stackrel{!}{=} 1\right)

We can easily calculate the Variance and Standard Deviation of the weights with respect to the overall domain (−r,+r)(-r,+r).

σ2(w)=12r∫−r+r(w−0)2dw=r23⇒σ(wi)=r3&σ(xi)=13\sigma^2(w)=\frac{1}{2 r} \int_{-r}^{+r}(w-0)^2 d w=\frac{r^2}{3} \Rightarrow \sigma\left(w_i\right)=\frac{r}{\sqrt{3}} \quad \& \quad \sigma\left(x_i\right)=\frac{1}{\sqrt{3}}

We then use the Strong Law of Large Numbers to calculate rr with

⇒σ(∑i=1nwixi)=!nσ(wixi)=nr3≈1\Rightarrow \quad \sigma\left(\sum_{i=1}^n w_i x_i\right) \stackrel{!}{=} \sqrt{n} \sigma\left(w_i x_i\right)=\sqrt{n} \frac{r}{3} \approx 1

and

r=3nr=\frac{3}{\sqrt{n}}

We should thus choose values of ww in (−r,+r)(-r,+r) to get good learning.

More Ressources