Derivatives Comparison Table

In multivariable calculus and functional analysis, various types of derivatives are used depending on the function's domain, codomain and application. This article summarizes and explains key derivative concepts such as the ordinary derivative, total derivative, gradient, Jacobian, Hessian, Gateaux derivative, and Fréchet derivative.

Derivative (Single-variable)

• Target Function:

  $$f:\mathbb{R}\to\mathbb{R}$$

• Formula:

  $$f^\prime(x) = \lim_{{h}\to{0}} \frac{f(x+h)-f(x)}{h} \in \mathbb{R}$$

• Meaning: Measures the instantaneous rate of change (slope) of a function with respect to a single variable.

Total Derivative

• Target Function: $f:\mathbb{R}^n \rightarrow \mathbb{R}$

• Formula: $df = \nabla f(\boldsymbol{x})^T d\boldsymbol{x} \in \mathbb{R}$

• Meaning: Gives the best linear approximation of a multivariable function. It corresponds to the dot product of the gradient and a small displacement.

Gradient

• Target Function: $f:\mathbb{R}^n \rightarrow \mathbb{R}$

• Formula: $\nabla f(\boldsymbol{x}) = \left[ \frac{\partial f}{\partial x_1}, ..., \frac{\partial f}{\partial x_n} \right]^T \in \mathbb{R}^n$

• Meaning: Vector that points in the direction of the steepest ascent.

Jacobian

• Target Function: $f:\mathbb{R}^n \rightarrow \mathbb{R}^m$

• Formula: $J(x)=[J_{ij}] = \left[\frac{\partial f_i}{\partial x_j} \right] \in \mathbb{R}^{m\times n}$

• Meaning: Generalization of the derivative to vector-valued functions. It contains all the first-order partial derivatives.

Hessian

• Target Function: $f:\mathbb{R}^n \rightarrow \mathbb{R}$

• Formula: $H(x)=[H_{ij}] = \left[ \frac{\partial^2 f}{\partial x_i \partial x_j}\right] \in \mathbb{R}^{n\times n}$

• Meaning: Matrix of second-order partial derivatives. It encodes the curvature of the function and is used in optimization.

Gateaux Derivative

• Target Function: (normed space) $F: X \rightarrow X$

• Formula: $D_G F(x; h) = \lim_{\epsilon \rightarrow 0} \frac{F(x + \epsilon h) - F(x)}{\epsilon} \in \mathbb{R}$

• Meaning: Directional derivative in the direction of a given vector $h$. It is a generalization of the derivative to infinite-dimensional spaces.

Fréchet Derivative

• Target Function: (Banach spaces) $F: X \rightarrow Y$

• Formula: $\lim_{\|h\| \rightarrow 0}\frac{\|F(x+h)−F(x)−A(h)\|}{\|h\|}=0$

• Meaning: Provides the best linear approximation in all directions. Stronger and more general than the Gateaux derivative.

Summary Table

Concept	Target Function	Meaning	Mathematical Expression
Derivative	$f:\mathbb{R} \rightarrow \mathbb{R}$	Measures the rate of change or slope of a single-variable function	$f^\prime(x) = \lim_{h \rightarrow 0} \frac{f(x+h)-f(x)}{h} \in \mathbb{R}$
Total Derivative	$f:\mathbb{R}^n \rightarrow \mathbb{R}$	Linear approximation of multivariable functions; expressed via the dot product with the gradient	$df = \nabla f(\boldsymbol{x})^T d\boldsymbol{x} \in \mathbb{R}$
Gradient	$f:\mathbb{R}^n \rightarrow \mathbb{R}$	Coefficient vector of the total derivative; indicates the direction of the steepest ascent	$\nabla f(\boldsymbol{x}) = \left[ \frac{\partial f}{\partial x_1}, ..., \frac{\partial f}{\partial x_n} \right]^T \in \mathbb{R}^n$
Jacobian	$f:\mathbb{R}^n \rightarrow \mathbb{R}^m$	Organizes all first-order partial derivatives into a matrix	$J(x)=[J_{ij}] = \left[\frac{\partial f_i}{\partial x_j} \right] \in \mathbb{R}^{m\times n}$
Hessian	$f:\mathbb{R}^n \rightarrow \mathbb{R}$	Organizes all second-order partial derivatives into a symmetric matrix; encodes curvature	$H(x)=[H_{ij}] = \left[ \frac{\partial^2 f}{\partial x_i \partial x_j}\right] \in \mathbb{R}^{n\times n}$
Gateaux Derivative	$F: X \rightarrow X$	Derivative in a specified direction $h$	$D_G F(x; h) = \lim_{\epsilon \rightarrow 0} \frac{F(x + \epsilon h) - F(x)}{\epsilon} \in \mathbb{R}$
Fréchet Derivative	$F: X \rightarrow Y$	Best linear approximation in all directions	$\lim_{\|h\| \rightarrow 0}\frac{\|F(x+h)−F(x)−A(h)\|}{\|h\|}=0$

Applications

In Optimization

• Gradient: Used in gradient descent algorithms
• Hessian: Used in Newton's method and second-order optimization

In Machine Learning

• Jacobian: Used in backpropagation for neural networks
• Gradient: Used in gradient-based optimization

In Functional Analysis

• Gateaux Derivative: Used in calculus of variations
• Fréchet Derivative: Used in optimization over function spaces

Key Relationships

1. Gradient and Total Derivative: The gradient is the coefficient vector in the total derivative expression
2. Jacobian Generalization: When $m=1$, the Jacobian reduces to the gradient (as a row vector)
3. Hessian as Jacobian: The Hessian is the Jacobian of the gradient
4. Fréchet ⊆ Gateaux: If the Fréchet derivative exists, then the Gateaux derivative exists and they are equal

Mathematical Prerequisites

Understanding these concepts requires familiarity with:

• Multivariable calculus
• Linear algebra
• Basic functional analysis (for Gateaux and Fréchet derivatives)

Practical Note

In most practical applications in engineering and machine learning, you'll primarily work with gradients, Jacobians, and Hessians. The Gateaux and Fréchet derivatives are more relevant in advanced mathematical analysis and optimization theory.