Vanilla Neural Networks

A neural network is just a math function: y = FNN(x).

FNN has a nested form. Think of it as a stack of layers. A 3-layer neural network that returns a scalar value looks like this:

y = FNN(x) = f₃(f₂(f₁(x)))

Each f — f₁, f₂, … fₙ — has the same form:

f(x) = g(Wx + b)

W (the weight matrix) and b (a bias vector) are the learned parameters, usually trained via gradient descent. g is the activation function, and it can be chosen differently for each layer.