Wednesday 12 June 2013

Information Theory

John R. Pierce, 1980 [1961]

Claude E. Shannon and Warren Weaver, 1949 [1998]
(June 2013)

One of these books is a very dense, technical work, heavy on the jargon and academic prose and littered with equations and diagrams that are very difficult for the layperson to get their head around. And the other is worse.

So I finally committed to reading the damn things. The Pierce book seems to be widely regarded as the standard overview and obviously the Weaver/Shannon volume is the Patient Zero of the whole sorry mess.

The maths started off as just about barely comprehensible and then very quickly accelerated away from my grasp. Conceptually the learning curve was initially less steep. The first few chapters of Pierce were testing but interesting and manageable; then he starts talking about multi-dimensional hyperspheres and writing perfectly cromulent sentences like the one below. Suddenly that learning curve turns into a cliff; the ground disappears and you’re left momentarily hovering in mid-air with your conceptual legs whirring away before realisation hits, leaving your intellect with barely enough time to look to camera and hold up a handpainted ‘Help!’ sign before the plummet commences.

“The noise figure is the ratio of the total output noise, including noise due to Johnson noise for a temperature of 293° Kelvin at the input and noise produced in the receiver, to the amplified Johnson noise alone.”

This is a quantum sentence. Dealing as Information Theory does with ideas of binary encoding and the extraction of meaning and order from chaos and entropy, it’s entirely appropriate that we should end up with sentences like this; a sentence that flips wholly between complete lucidity and utter incomprehensibility with each successive rereading without passing through any intermediate states in between. Clearly the only sensible thing to do is to shut it in a box with a radioactive trigger and some poison and never open the lid again. Or at least until you’re sure it’s dead…

I shall, in due course, be putting up some more constructive thoughts over at the other place, but it’ll take a while to boil them all down into anything even approaching coherence or applicability. Like the cat, I wouldn’t hold your breath.


  1. I once had a seminar course in Hegel's "Phenomenology of Mind"... As an undergrad you just figure you are stupid when you read that; however, once I read secondary works on it, I realized he was a spectacularly bad writer, with some interesting but wrong-headed notions. Lesson? Read a precis when the going gets tough. Life's too short.

    1. Interestingly, I've been reading almost the exact opposites of precis(es?) here, in that as each volume (read a couple of other books on this a while back) has got less and less pop-science and more technical, they've been getting shorter and shorter. Makes you realise just how dense the original was. In case that wasn't immediately obvious just from looking at it. Although otherwise you make a very valid point.

  2. Hey, nice!... Signals and noise sounds like you'll be dealing with Fourier transforms and such. Good stuff!

    1. Thought this might be up your street. Yep, they feature, but I'm more taken with the possibilities of Zipf's Law and Markov chains. I instinctively feel they could be quite powerful ways of explaining important things about language, but can't quite nail it down to specifics, or at least ones I can actually use.

      Of course it's just as likely I'm being seduced simply because they have funny foreign names.

  3. I know how you feel as I was in the same position a few years back. Certainly Pierce and Warren Weaver's intro to Shannon's original work are much more accessible than Shannon's work by itself depending on your level of mathematical sophistication. Read them a couple of times while skimming through the equations. Then take out a pencil and paper and work your way through some of the equations. This will make things infinitely easier going as most (if not all) mathematics isn't meant to be read quickly and easily like a novel.

    If it helps, Khan Academy has some intro videos on information theory:

    1. Thanks for the comment and the pep-talk.

      I'll be honest, the maths is way down the road at the moment. At the moment It's all about how I can make the ideas work for me. There may come a point where I need to plug in some numbers but that's probably a long, long way off.