Challenge: Shannon and Zipf

For those who don't know what Shannon Entropy and Zipf plots are they are ways to measure the amount of information in a signal or system.

Shannon Entropy:
https://en.wikipedia.org/wiki/Entropy_(information_theory)
https://www.youtube.com/watch?v=_PG-jJKB_do

Zipf's Law:
https://en.wikipedia.org/wiki/Zipf%27s_law
https://www.youtube.com/watch?v=fCn8zs912OE

Now given any file or stream of information it should be possible to determine the amount of information contained and if it's intelligent information.

Some examples:

If we do coin flips the information of each flip is one bit, this is an entropy of 1, but the Zipf plot for a given length of characters well it should be evenly distributed. If we count the number of 1s vs the number of 0s they should be about equal. 11, 10, 01, and 00 should all be about too. How about 111, 110, 101, 100, 011, 010, 001, and 000 again equal. This is because while there's information the information is unintelligent randomness.

The same would be (I'd think) true of Pi or e, looking digit-wise, or at the binary equivalent, the information each bit gives is effectively random.

However, inputting say the complete works of Shakespeare, well even I didn't know english there'd be some patterns in there that I could recognise and so show that the entropy is less than 1. Looking at the stream bit-wise letters start with common sequences 010 for capitals and 011 for lower case, so we know these sequences will be higher, and looking byte-wise/character-wise we know the english language has a Zipf plot, So we can say that though there's less information per bit the information is more intelligent/less random.

And there are some types of information that're denser and at least as intelligent as english, compressed data for example should have the same Zipf as it's source data but higher Shannon score or the same might be said for txt tlk wch ignors cmmn wrds shrtns othrs n uses shrthnds lyk lol or emoji fittin big concepts in fwr chars. And other data might be less dense and more predictable, I'd think image data could fall in this camp.

So the challenge would be to make a program that takes an input and evaluate how information dense it is and how intelligent that information is, either by giving it a score or displaying the Shannon and Zipf data,

I think the two functions work well together because they're both different ways of analysing the same type of data

Shannon: Given a bit string of length N how predictable is bit N+1
Zipf: How common is a given bit string of length N.

Each would require taking an input file and dividing it up, the reason why I'm leaning to bit strings is that I think it'd be interesting to see how information dense other things are than text say pictures, compressed files, even coded messages. Then maybe look at different lengths of bit strings, 1, 2, 4, 8, 12, 16, 24, 32, might be good starting choices, but it'd be interesting to see if this could be done more dynamically, the user choosing to scroll through from single bit scores through to the limits of processing, maybe with it being dynamic it reads and processes only a limited amount more per frame, input MBs of data and it starts at the beginning building up a picture of how Zipfian and predictable the data is 1024 bits per frame