Numerical differentiation is a method used to approximate a derivative using finite perturbation differences.
There are three basic methods for numerical differentiation: Forward, Backward, and Central difference methods.
It can be used when you don't have the derived function but need to perform gradient-based optimization.
In a computer, if the finite perturbation value is too small,
the rounding error may become significant, leading to inaccurate results in the approximation of the derivative.
( \( 10^{-4} \sim 10^{-6} \) )
Forward difference method (전방 차분)
The forward difference method approximates the derivative of a function
by using the value of the function at the current point and a small step forward.
\[
\,\\
f'(x) = \lim_{h\rightarrow0} \frac{f(x + h) - f(x)}{h}
\,\\
\]
Backward difference method (후방 차분)
The backward difference method approximates the derivative of a function
by using the value of the function at the current point and a small step backward.
\[
\,\\
f'(x) = \lim_{h\rightarrow0} \frac{f(x - h) - f(x)}{h}
\,\\
\]
Central difference method (중앙 차분)
The central difference method approximates the derivative
by using both the forward and backward steps around the point,
which can give a more accurate estimate.
\[
\begin{align*}
\,\\
f'(x) &= \lim_{h \, \rightarrow \, 0} \, \frac{1}{2} \cdot \left( \frac{f(x + h) - f(h)}{h} - \frac{f(x - h) - f(h)}{h} \right)
\\\,\\
&= \lim_{h \, \rightarrow \, 0} \, \frac{1}{2} \cdot \frac{f(x + h) - f(x - h)}{h}
\\\,\\
&= \lim_{h \, \rightarrow \, 0} \, \frac{f(x + h) - f(x - h)}{2h}
\,\\
\end{align*}
\]