Shannon Entropy Calculator: Information in Bits from Probabilities
The Shannon entropy in bits of a set of probabilities, with the maximum and the efficiency.
Engineering student calculators
Free textbook calculators with every step shown, so you can check your own working. Pick one, type in your numbers and see the formulas worked out. Or switch to Ask a question for an explained answer. No account needed.
Calculated from standard textbook formulas, with every step shown. The formulas are fixed, so the working does not need checking: what matters is the numbers you type in. Use these calculators only if you are competent to choose the inputs (loads, material strength, dimensions, supports) and check your units and assumptions. For learning and checking your own working: design and sign-off are for a qualified engineer. Drafting Central does not store the numbers you type.
How this calculator works
Enter the probability of each outcome (or the counts, which are turned into probabilities). The entropy is minus the sum of each probability times its base-2 logarithm.
It gives the entropy in bits, the largest entropy for that many outcomes (when they are all equally likely), the efficiency and the redundancy. A fair coin has 1 bit; four equally likely outcomes have 2 bits.
The entropy is the smallest average number of bits per symbol that a perfect code could use, so it limits how far a message can be compressed.
Formulas used
- H = -\sum_i p_i\log_2 p_i
- H_{max} = \log_2 n
Worked example
With the default numbers in the calculator above:
- Outcomes with a chance above zero: 4
- Shannon entropy: 1.75 bits
- Largest entropy for this many outcomes (all equally likely): 2 bits
- Efficiency: 87.50 %
- Redundancy: 12.50 %
Change the numbers in the calculator to see the full working, every step, for your own case.
Common questions
What is a bit of entropy?
The information in one fair coin toss.
When is the entropy largest?
When all the outcomes are equally likely.
What is redundancy?
How far the entropy is below its maximum. It is the part that a code can squeeze out.
Related calculators
Got a question about this? Ask an engineering question and get a plain-language answer with references.
Further reading: Shannon (1948), A Mathematical Theory of Communication; Cover and Thomas, Elements of Information Theory.
Need a calculator we do not have?
Embed this calculator on your own page
Copy this code into your page. The calculator runs inside the frame and links back to Drafting Central.
Need this drawn, designed or checked?
Drafting Central is a mechanical engineering and drafting business working with customers across Australia and around the world. A calculator gives you the numbers; we can turn them into drawings and designs.
Need help with a real project?
Talk to Drafting Central →