Derivative as a little-o approximation

Statement

Let $V$ and $W$ be normed vector spaces over $\mathbb{R}$, let $U \subseteq V$ be open, and let $f \colon U \to W$. Let $x \in U$, and let $A \colon V \to W$ be a bounded linear map. Then $A$ is the Fréchet derivative of $f$ at $x$ if and only if $$f(x + h) = f(x) + A h + r(h), \qquad \text{where } r(h) = o(\|h\|) \text{ as } h \to 0.$$ Here $r(h) = o(\|h\|)$ means that for each $\varepsilon > 0$ there is a $\delta > 0$ such that $\|r(h)\| \le \varepsilon \|h\|$ for each $h$ with $\|h\| < \delta$ and $x + h \in U$.

Remarks

The remainder $r(h)$ is defined by the equation, so the statement says that the error of the linear approximation tends to $0$ faster than $\|h\|$. The little-o form includes $h = 0$, where $r(0) = 0$. This is why it uses $\le$ and does not divide by $\|h\|$. This form is often the most convenient one for proofs, for example of the chain rule.

Explanations

L1 human

The equation splits $f(x + h)$ into three parts: the value at $x$, a linear part in $h$, and an error. The derivative condition says only that the error is tiny compared with the step: halving the step more than halves the error.