information theory - Can the Entropy Be Used as a Measure of the Contrast of an Image?

I was looking for value that can measure a contrast of an image. And I found a couple of answers like this one https://stackoverflow.com/questions/13397394/how-to-measure-contrast-in-opencv-visual-c/13398126.

But I can't find any explanations WHY is that. No articles or links. So I wrote this code:

def shannon_entropy(img):

    histogram = img.histogram()
    histogram_length = sum(histogram)

    samples_probability = [float(h) / histogram_length for h in histogram]

    return -sum([p * math.log(p, 2) for p in samples_probability if p != 0])

And found the Shannon entropy for four images:

- original image "image_1". Entropy is 7.82426730799067

- more contrast image "contrast". Entropy is 5.682661230050924

- much more contrast image "contrast2". Entropy is 2.1507540571698587

- less contrast image "contrastLow". Entropy is 6.565885023104716

I thought, that for the last image entropy will be the highest, but no. And it completely baffled me.

Can somebody explain:

1. Why entropy is a measure of contrast?


2. What's wrong with my images or calculations?

Answer

Why entropy value can be used as measure of contrast of an image?

It can't. Entropy is not a function of contrast.

I'll show why. Time for a bit of basic Information Theory¹:

Definitions

Let's define $H(·)$ as the entropy of a source; imagine $X$ is your source picture, $Y$ is the picture after you subjected it to some operation $F:\, F(X)=Y$ (e.g. contrast spreading, or compression, or adding random noise).

Intuition of Entropy

$H(X)$ is a measure for the amount of information in the source image. It's hence a measure for the "uncertainty" of the value of each pixel; if that picture was just one intensity all over, the uncertainty would be low, and so the information, and thus the entropy.

Let's assume your input image has enough contrast to at least make use of 8 bits of dynamic range. Then, the highest entropy you could get would be 8 bit per pixel – in case all pixel values are present equally likely. That's how I know your entropy calculation is broken – a black and white image only has two values, and the discrete uniform distribution over 2 values has exactly 1 bit of entropy, and any other distribution has less. You'll need to fix your entropy calculation!

Contrast

However, contrast doesn't care about the full distribution – there's different contrast definitions/measures, but they typically only consider the first and second statistical moment. Many distributions can have the same variance – but only one is the uniform distribution that maximizes entropy. From that alone you can see that while there is a relation between contrast and entropy, it's not a direct relation:

You just can't have high entropy with very low contrast. However, high contrast doesn't say much about entropy.

Applying image transformations

Consider your contrast enhancements:

$H(X|Y)$ is the amount of the uncertainty that remains about the original image when you apply the operation $Y=F(X)$.

When your operation $F$ is reversible (e.g. if you just invert the pixel values, or if you scale them by 2 without clipping because an original value was larger than half the maximum pixel value), then there's nothing unclear about the original image (just apply the "reverse" operation), and thus $H(X|F(X)=Y)=0$. If you think about it, it's pretty intuitive: A revertible operation doesn't change the probabilities, it just changes the values that have these probabilities.

Information in the face of change

We call the information we get about the source when looking at our transformed image the mutual information between $X$ and $Y$, $I(X;Y)$. It's the difference between the source $X$'s entropy and the conditional entropy $H(X|Y)$, i.e. $I(X;Y) = H(Y) - H(Y|X)$!

In the case that the operation is revertible, as you can see, $H(X|Y)=0\implies I(X;Y)=H(X)$.

And, since we know the deterministic operation $F$, the uncertainty about what $Y$ looks like if we know $X$ is $H(Y|X)=0$, too, so that $H(Y)=I(X;Y)$, and conclusively, $H(Y)=H(X)$. In other words, when you do an invertible operation, you can't increase or reduce entropy.

Now, there's the thing: Your contrast operations aren't revertible, simply because

1. your image has finite bit depth, so some rounding happens and different values in $X$ get mapped to the same value in $Y$,


2. your image has finite bit depth, so there is some clipping when you reach the extreme values.

That means that you get a $H(X|Y) > 0$; from that we can see that $I(X;Y). However, since the operation you apply is still deterministic (i.e. you can calculate $Y=F(X)$ based on $X$, it's always the same), the uncertainty about $Y$ knowing $X$ is zero, $H(Y|X) = 0$, and therefore

$$H(Y)=I(X;Y)+H(Y|X)=I(X;Y) + 0 = I(X;Y) < H(X),$$

which is exactly what you see: all your contrast-stretching and -reducing operations decrease the entropy compared to the original image.

1. Having high contrast is a necessary, not a sufficient condition for high entropy


2. Entropy is limited by the number of values occupied


3. Entropy is invariant to revertible (that implies deterministic) (image) operations


4. Entropy is reduced by deterministic non-revertible operations

Corollary:

5. if you want to increase an image's entropy, you'll need to do something non-deterministic, like adding noise.


6. Information theory is fun and you should definitely study it.

Here's an entropy function that makes sense; I don't know what kind of data type img is, but I assume you can just go through all its values (it's iterable).

from collections import Counter

def shannon_entropy(img):

    ld = lambda x: math.log(x, 2)
    value_counter = Counter(img)
    N = len(img)
    H = 0
    for count in value_counter.values(): #this gives the counts of values 
        p = float(count)/N
        H = H - p*ld(p)
    return H

¹ yes, that is a pun.