Hyperbolic Tangent
The tanh analyzes diversity in the input space similar to SVMs. This scales well to high dimensional spaces where tanh does a multidimensional diversity analysis.
y & =\tanh \left(\sum_i w_i x_i-w_0\right) \\ & =\tanh (w \cdot(x-b)) \quad \text { Hessian normal form } \\ & =\tanh (\|w\| \cdot\|x-b\| \cdot \cos \varphi) \\ & =\tanh (\|w\| \cdot \operatorname{dist}(x, \text { hyperplane })) \rightarrow \pm 1 \end{aligned}$$ Here we use the Hessian Normal Form to rewrite the sum into a dot product. We can calculate $b$ to be $\frac{w_0}{w_i}$. Then we can use the normal dot product rule to rewrite it into the product of the two magnitudes multiplied by the angle between them. Here we can first see that the result of the neuron depends both on the distance to the bias and the angle between the weight and the input. Geometrically we can view it like this: ![[CleanShot 2023-11-18 at 16.05.10@2x.png]] Multiple dimensions:y=\sum_j v_j \cdot \tanh \left(\sum_i w_{j i} x_i-w_{j 0}\right)
y=\sum_j v_j \tanh \left(\sum_i w_{j i} x_i+\mathrm{u}{\mathrm{ji}} \mathrm{x}{\mathrm{i}}^2-w_{j 0}\right)
when $u$ is not zero the network can also model with [[Radial Basis Function Kernel|RBFs]]. ## First Derivative\tanh ^{\prime}(\text { netin })=1-(\tanh (\text { netin }))^2
## Simplifications The following simplification is an easy to compute approximation of the tanh function. But it is non-differentiable in two points and we multiply the error flow with zero when leaving the middle range. In [[Backpropagation]] we have\left.\partial=f^{\prime}(\text { netin }) d e v\right)
f(x)=\tanh (x) \quad f_1(x)=\frac{x}{1+|x|} \quad f_2(x)=\frac{x}{\sqrt{1+x^2}}