Sinus Activation

We can approximate the tanh function with the non-monotonic sinus function.

CleanShot 2023-11-18 at 22.57.43@2x.png

We can rewrite multiplications of sinus functions into sums of sinus functions

sin⁡(x)⋅sin⁡(y)=12[sin⁡(x−y+π2)+sin⁡(x+y−π2)]\sin (x) \cdot \sin (y)=\frac{1}{2}\left[\sin \left(x-y+\frac{\pi}{2}\right)+\sin \left(x+y-\frac{\pi}{2}\right)\right]

which is a nice idea as sums are easier to compute than multiplications.

Also, sinus partly avoids vanishing Gradients in Backpropagation.