Rectified Linear Unit
An Activation Function, mainly used in Image Recognition.
x & & x \geq 0 \\
& \text { if } & \\
0 & & x<0
\end{array}\right\}$$
## Smoothed
f(x)=\frac{1}{p} \ln \left(1+e^{p x}\right)
with[[LogisticFunction]]as[[Gradient]]
f^{\prime}(x)=\frac{1}{1+e^{-p x}}.
∗∗Pros∗∗−doesnotsqueezeerrorflowinbackwardpass−cancompletelyblockerrorflowwhenvaluesarenegativeinbackwardpass(helpfulforimagerecognition?)∗∗Cons∗∗−explodingvaluesinforwardpass−notdifferentiableatzero−doesnotsqueezeerrorflowinbackwardpass