Fast sigmoid functions

2013-03-29

Wikipedia lists quite a few sigmoid functions. Which one should be used? In some problem domains the choice of the sigmoid function doesn't really matter. And in this case you may prefer the one which is easier faster to calculate.

Just a quick benchmark script sigmoid-bench.c, and now I have a good reason to prefer some sigmoid functions over the others.

% gcc -Wall -O2 -lm -o sigmoid-bench{,.c} -std=c99 && ./sigmoid-bench
atan(pi*x/2)*2/pi   24.1 ns
atan(x)             23.0 ns
1/(1+exp(-x))       20.4 ns
1/sqrt(1+x^2)       13.4 ns
erf(sqrt(pi)*x/2)    6.7 ns
tanh(x)              5.9 ns
x/(1+|x|)            5.4 ns

That's on a mobile Core i5 CPU, with GCC 4.7.2 compiler. The last digit is not reliable, but overall the picture is clear.

The first place is:

$$\frac{x}{1 + |x|},$$

and the runner up is the hyperbolic tangent:

$$\tanh x.$$

Error function erf, which comes the third, is available since C99 (effectively everywhere but in Microsoft compilers). The arctangent is probably to avoid unless it's an organic part of your problem.