Entropy is a concept which originally arose from the study of the physics of heat engines. It can be described as a measure of the amount of disorder in a system. An organized structure, such as a crystal or a living organism, is very highly ordered and consequently has low entropy. When the crystal is heated sufficiently, it melts and becomes liquid, a much less ordered state. When the organism dies, it decays and becomes completely disrupted. In either system, its entropy increases.

Another way of expressing entropy is to consider the spread of states which a system can adopt. A low entropy system occupies a small number of such states, while a high entropy system occupies a large number of states.

In the case of an image, these states correspond to the gray levels which the individual pixels can adopt. For example, in an 8-bit pixel there are 256 such states. If all such states are equally occupied, as they are in the case of an image which has been perfectly histogram equalized, the spread of states is a maximum, as is the entropy of the image. On the other hand, if the image has been thresholded, so that only two states are occupied, the entropy is low. If all of the pixels have the same value, the entropy of the image is zero.

Note that, in this progression, as the entropy of the image is decreased, so is its information content. We moved from a full gray scale image, with high entropy, to a thresholded binary image, with low entropy, to a single-valued image, with zero entropy.

Information, in this context, refers to the announcement of the unexpected.
If the pixels of an image were inspected, and found to be the same. This information
could have been communicated in a very short message. The information content is
said to be low simply because it *can* be communicated in a short message.
If the pixels are changing in unexpected ways, however, longer messages are required
to communicate this fact and the information is said to increase. This assumes,
of course, that all changes in the image are meaningful. Changes due to noise are
still considered to be information in that they describe the image as it actually
is, rather than as it should be.

Noise and heat play similar roles in increasing the
entropy of systems.
The *entropy H* of an image is defined as

where *M* is the number of gray levels and *p _{k}* is the probability
associated with gray level

Maximum entropy is achieved in the case of a uniform probability distribution. If
, then *p _{k}* is constant and given by

Minimum entropy is achieved when the image itself is constant, that is, all of the pixels have
the same gray level *k*. For that gray level, *p _{k}* = 1, and

Entropy sets a lower bound on the average number of bits per pixel required to encode an image without distortion. This is only true, however, for uncorrelated images. Consider the images below,

- Rhys Lewis,
*Practical Image Processing*, Ellis Horwood Limited, 1990. ISBN 0-13-6383525-0. p 90-91. - William Pratt,
*Digital Image Processing, Second Edition*, John Wiley & Sons, Inc., 1991. ISBN 0-471-85766-1. p. 561-563. - Azriel Rosenfeld and Avinash Kak,
*Digital Picture Processing, Second Edition, Volume 1*, Academic Press, 1982. ISBN 0-12-597301-2. p. 194-198.

Maintained by John
Loomis, last updated *10 June 1998 *