Long Short Term Memory Network

A special form of Recurrent Neural Network that works well with long-range dependencies in the sequences. It also reduces the risk of vanishing or exploding Gradients.

Intuition It adds longer lasting memory to a Recurrent Neural Network.

$s_t={\mathrm{switch}}_1\left(u_t\right)\cdot s_{t-1}+{\text{ switch }}_2\left(u_t\right)\cdot \tanh \left(A{s}_{t-1}+B{u}_t\right)$

A more efficient way by an Exponential Smoothing embedding. $s_t=(1-D)s_{t-1}+D\tanh \left(A{s}_{t-1}+B{u}_t\right)=s_{t-1}+D\left(\tanh \left(A{s}_{t-1}+B{u}_t\right)-s_{t-1}\right)$

The matrix $D$ can be learned by the model. It can thus decide on its own how much long term memory it needs.

Feature Selection

Feature Selection using Sensitivity Analysis

Resources

http://colah.github.io/posts/2015-08-Understanding-LSTMs/