State Space Model

If we interpret a dynamical system as a state space model we can think of continous functions $\frac{ds(t)}{dt}=f(s(t),u(t))$ , the state transition function and $y(t)=g(s(t))$ a function that observes the current state and returns an Observation.

• $u$ → external factors
• $s$ → state
• $y$ → observations

If $u$ experiences some kind of shock, then this will have the possibility to first propagate a few times through the network via $s$ before it becomes visible in $y$. In the original Feed-forward Neural Network it would directly affect the outputs.

RNN

One can model a State Space Model using RNNs in two ways:

• RNN Backward Approach
• RNN Forward Approach

When there is only a small timespan between individual time steps, then the two approaches will be quite similar.