Information Theory Quotes

Chaos is merely order waiting to be deciphered

My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'

Information consists of differences that make a difference.

While knowledge is orderly and cumulative, information is random and miscellaneous.

The importance of information is directly proportional to its improbability.

All information is everywhere. Information that is not here is nowhere.

Any sufficiently crisp question can be answered by a single binary digit-0 or 1, yes or no.

During my Ph.D. program, I became interested in the informational structure of markets that turned into the work on signaling, which was the part of my early work that was recognized for the Nobel Prize, but it was not really a subject at the time.

Entropy shakes its angry fist at you for being clever enough to organize the world. (p 2)

Entropy and optimism: the twin forces that make the world go around.

A fundamental principle of information theory is that you can’t guarantee outcomes… in order for an experiment to yield knowledge, it has to be able to fail. If you have guaranteed experiments, you have zero knowledge

Modern statisticians are familiar with the notion that any finite body of data contains only a limited amount of information on any point under examination; that this limit is set by the nature of the data themselves, and cannot be increased by any amount of ingenuity expended in their statistical examination: that the statistician's task, in fact, is limited to the extraction of the whole of the available information on any particular issue.

Communications without intelligence is noise. Intelligence without communications is irrelevant.

It from bit symbolizes the idea that every item of the physical world has at bottom...an immaterial source and explanation...that all things physical are information-theoretic in origin and that this is a participatory universe.

The body itself is an information processor. Memory resides not just in brains but in every cell. No wonder genetics bloomed along with information theory. DNA is the quintessential information molecule, the most advanced message processor at the cellular level - an alphabet and a code, 6 billion bits to form a human being.

Orderliness by itself is not sufficient to account for the nature of organized systems in general or for those created by man in particular. Mere orderliness leads to increasing impoverishment and finally to the lowest possible level of structure, no longer clearly distinguishable from chaos, which is the absence of order. A counterprinciple is needed, to which orderliness is secondary. It must supply what is to be ordered.

Every force field is simultaneously a field of information because even physics now acknowledges that an atom is not only a hierarchy of different states of energy, or different states of force fields.

You're an entropic chaos factor

The Web is now philosophical engineering. Physics and the Web are both about the relationship between the small and the large.

There is no effective difference between guessing a variable that is not random, but for which information is partial or deficient, and a random one. In this sense, guessing (what I don't know, but what someone else may know) and predicting (what has not taken place yet) are the same thing.

I love computer programmers. They have a very beautiful definition of complexity as 'the capacity to transmit the maximum information with the minimum data'.

I believe that consciousness is the way information feels when being processed.

[With quantum computers] you can calculate how many bits are in the universe, how much energy it takes to flip them, how much energy exists, and use that to rule out lots of things about the universe's history. Anything that takes more bit flips couldn't have happened.

At its most fundamental, information is a binary choice. In other words, a single bit of information is one yes-or-no choice.

Money is information, like bits.