Summed area table

A summed area table is a data structure and algorithm for quickly and efficiently generating the sum of values in a rectangular subset of a grid. In the image processing domain, it is also known as an integral image. It was first introduced to computer graphics in 1984 by Frank Crow for use with mipmaps. In computer vision it was first prominently used within the Viola–Jones object detection framework in 2001. However, historically, this principle is very well known in the study of multi-dimensional probability distribution functions, namely in computing 2D (or ND) probabilities (area under the probability distribution) from the respective cumulative distribution functions.

The algorithm

As the name suggests, the value at any point (xy) in the summed area table is just the sum of all the pixels above and to the left of (xy), inclusive:

$I(x,y)=\sum _{\begin{smallmatrix}x'\leq x\\y'\leq y\end{smallmatrix}}i(x',y')$ Moreover, the summed area table can be computed efficiently in a single pass over the image, using the fact that the value in the summed area table at (xy) is just:

$I(x,y)=i(x,y)+I(x-1,y)+I(x,y-1)-I(x-1,y-1)\,$ Once the summed area table has been computed, the task of evaluating any rectangle can be accomplished in constant time with just four array references. Specifically, using the notation in the figure at right, having A=(x0, y0), B=(x1, y0), C=(x0, y1) and D=(x1, y1), the sum of $i(x,y)$ over the rectangle spanned by A, B,C and D is just

$\sum _{\begin{smallmatrix}x0 Extensions

• This method is naturally extended to continuous domains.
$\sum _{p\in \{0,1\}^{d}}(-1)^{d-\|p\|_{1}}I(x^{p})\,$ • This method has been extended to high-order integral image as in the work of Phan et al.  who provided two, three, or four integral images for quickly and efficiently calculating the standard deviation (variance), skewness, and kurtosis of local block in the image. This is detailed below:

To compute variance or standard deviation of a block, we need two integral images:

$I(x,y)=\sum _{\begin{smallmatrix}x'\leq x\\y'\leq y\end{smallmatrix}}i(x',y')$ $I^{2}(x,y)=\sum _{\begin{smallmatrix}x'\leq x\\y'\leq y\end{smallmatrix}}i^{2}(x',y')$ The variance is given by:

$\operatorname {Var} (X)={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-\mu )^{2}.$ $\operatorname {Var} (X)={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}^{2}-2\cdot \mu \cdot x_{i}+\mu ^{2})={\frac {1}{n}}\sum _{i=1}^{n}((x_{i})^{2})-2\cdot \sum _{i=1}^{n}(\mu \cdot x_{i})+\sum _{i=1}^{n}(\mu ^{2}))={\frac {1}{n}}\sum _{i=1}^{n}((x_{i})^{2})-2\cdot \sum _{i=1}^{n}(\mu \cdot x_{i})+n\cdot (\mu ^{2}))$ $={\frac {1}{n}}\sum _{i=1}^{n}((x_{i})^{2})-2\cdot \mu \cdot \sum _{i=1}^{n}(x_{i})+n\cdot (\mu ^{2}))={\frac {1}{n}}(S_{2}-2\cdot S_{1}/n\cdot S_{1}+n\cdot ((S_{1}/n)^{2}))={\frac {1}{n}}(S_{2}-(S_{1})^{2}/n)$ Similar to the estimation of the mean ($\mu$ ) and variance ($Var$ ), which requires the integral images of the first and second power of the image respectively (i.e. $I,I^{2}$ ); manipulations similar to the ones mentioned above can be made to the third and fourth powers of the images (i.e $I^{3}(x,y),I^{4}(x,y)$ .) for obtaining the skewness and kurtosis. But one important implementation detail that must be kept in mind for the above methods, as mentioned by F Shafait et al.  is that of integer overflow occurring for the higher order integral images in case 32-bit integers are used.