Neural Network

Basic Neural Network architecture

• Weights
• Bias Weight
  • needed to allow for general hyperplanes that do not necessarily go through the origin
  • in RNNs a missing bias weight can be corrected by one of the internal state neurons.
• Activation Function
• Backpropagation

A neural network can always be expanded by the concatenation of mutliple layers in contrast to Taylor Expansion where complexity is increased with additive terms.

Universal Approximation Theorem

Existence Theorem 3-layer neural nets can approximate any continous function on a compact domain.

For example:

• Feed-forward Neural Network
• Deep Neural Network

Specialized Networks

• Recurrent Neural Network
  • Long Short Term Memory Network
  • ECNN
  • HCNN