Critical speed

The softmax activation function is a neural transfer function. In neural networks, transfer functions calculate a layer's output from its net input.

On the other hand, a paper by Cadieu et al. argues that it is a biologically plausible approximation to the maximum operation. The authors have used it to simulate an invariance operation of complex cells where they defined it as

$y=g\left(\frac{\sum _{j=1}^n{x}_j^{q+1}}{k+\left(\sum _{j=1}^n{x}_j^q\right)}\right)\text{,}$

where g is a sigmoid function, x represents the value of input nodes, k a small constant to avoid division by zero, and the exponent q a parameter to control the non-linearity.

Artificial neural networks

In neural network simulations, the term softmax activation function refers to a similar function defined by

$\sigma :{\mathbb{ℝ}}^n\times \mathbb{\mathbb{N}}\to \mathbb{ℝ}$

$\sigma (\mathbf{\text{q}},i)=\frac{\exp (q_i)}{\sum _{j=1}^n\exp (q_j)}\text{,}$

where the vector q is the net input to a softmax node, and n is the number of nodes in the softmax layer. It ensures all of the output values are between 0 and 1, and that their sum is 1. This is a generalization of the logistic function to multiple variables.

Since the function maps a vector and a specific index i to a real value, the derivative needs to take the index into account:

$\frac{\partial }{\partial q_k}\sigma (\mathbf{\text{q}},i)=\dots =\sigma (\mathbf{\text{q}},i)({\delta }_{ik}-\sigma (\mathbf{\text{q}},k))$

Here, the Kronecker delta is used for simplicity (cf. the derivative of a sigmoid function, being expressed via the function itself).

See Multinomial logit for a probability model which uses the softmax activation function.

Reinforcement learning

In the field of reinforcement learning, a softmax function can be used to convert values into action probabilities. The function commonly used is:

$P_t(a)=\frac{\exp (q_t(a)/\tau )}{\sum _{i=1}^n\exp (q_t(i)/\tau )}\text{,}$

where the action value $q_t(a)$ corresponds to the expected reward of following action a and $\tau$ is called a temperature parameter (in allusion to chemical kinetics). For high temperatures ($\tau \to \mathrm{\infty }$), all actions have nearly the same probability and the lower the temperature, the more expected rewards affect the probability. For a low temperature ($\tau \to 0^{+}$), the probability of the action with the highest expected reward tends to 1.

Smooth approximation of maximum

When parameterized by some constant, $\alpha >0$, the following formulation becomes a smooth, differentiable approximation of the maximum function:

${\mathcal{𝒮}}_{\alpha }\left({\left\{x_i\right\}}_{i=1}^n\right)=\frac{\sum _{i=1}^n{x}_i{e}^{\alpha x_i}}{\sum _{i=1}^n{e}^{\alpha x_i}}$

${\mathcal{𝒮}}_{\alpha }$ has the following properties:

1. ${\mathcal{𝒮}}_{\alpha }\to \max$ as $\alpha \to \mathrm{\infty }$
2. ${\mathcal{𝒮}}_0$ is the average of its inputs
3. ${\mathcal{𝒮}}_{\alpha }\to \min$ as $\alpha \to -\mathrm{\infty }$

The gradient of softmax is given by:

${\mathrm{\nabla }}_{x_i}{\mathcal{𝒮}}_{\alpha }\left({\left\{x_i\right\}}_{i=1}^n\right)=\frac{e^{\alpha x_i}}{\sum _{j=1}^n{e}^{\alpha x_j}}[1+\alpha (x_i-{\mathcal{𝒮}}_{\alpha }\left({\left\{x_i\right\}}_{i=1}^n\right))]\text{,}$

which makes the softmax function useful for optimization techniques that use gradient descent.

Softmax transformation

Template:Section OR The softmax function is also used to standardize data which is positively skewed and includes many values around zero.Potter or Ceramic Artist Truman Bedell from Rexton, has interests which include ceramics, best property developers in singapore developers in singapore and scrabble. Was especially enthused after visiting Alejandro de Humboldt National Park. It will take a variable such as revenue or age and transform the values to a scale from zero to one. This type of data transformation is needed especially when the data spans many magnitudes.

For example, revenues for customers could span anywhere from 0 to 300.000. Let's say we have a range of revenue numbers between 3 and 300.000. If these numbers are expressed in powers of 10, then 3 becomes $3\times 10^0$ and 300.000 becomes $3\times 10^5$. The number 10 when raised to the power 0 becomes 1, so these two numbers expressed as powers of 10 span 5 orders of magnitude. Range scaling is a typical reason to use a function such as softmax.Potter or Ceramic Artist Truman Bedell from Rexton, has interests which include ceramics, best property developers in singapore developers in singapore and scrabble. Was especially enthused after visiting Alejandro de Humboldt National Park.

External links

• description of softmax as a substitute for maximum by John D. Cook