Partial Derivatives, Reverse-Mode Autodiff
A partial derivative measures how much a function changes when you nudge one input, holding every other input fixed. For f(x, y) = x²y + y + 2, the partial derivative with respect to x asks: if y stays put, how fast does f move as x moves?
Manual differentiation
Mathematically, we know that ∂f/∂x = 2xy and ∂f/∂y = x² + 1, using a handful of rules:
- the derivative of a constant is
0 - the derivative of
axisa - the derivative of
x^aisa·x^(a-1) - the derivative of a sum is the sum of the derivatives:
(u + v)' = u' + v' - the derivative of a product follows the product rule:
(u·v)' = u'v + uv'
But how can a program know that?