Spine · Information and meaning
A spine traces the arc of a field — how it arrived at its present position.
Shannon made communication measurable without measuring meaning. Semantics formalized meaning. Pragmatics situated meaning in use and intention. Distributional semantics found statistical traces of meaning in use. Embeddings converted those traces into geometry. Transformers made the geometry contextual and dynamically compositional.
Stages are shown in conceptual order. True chronology diverges — Frege precedes Shannon by more than fifty years, and Harris precedes Grice. These traditions coexist rather than replace one another.
What this is
This is a spine, not a claim that one field simply replaced the previous field. These traditions ask different questions, use different units of analysis, and continue to coexist.
The evolution from information theory to contextual representation should not be read as a linear solution to meaning, but as a sequence of abstractions that progressively incorporated source uncertainty, denotation, compositionality, situated use, corpus distribution, vector geometry, and context-dependent prediction.
The spine at a glance
| Era | Primary question | Object modeled |
|---|---|---|
| Before Shannon | How much can this channel carry? | Signals and bandwidth |
| Shannon, 1948 | How much uncertainty does a message remove? | Source probability distributions |
| Semantics | What does an expression denote, and when is it true? | Denotation and truth conditions |
| Pragmatics | What did the speaker mean, here, by saying this? | Intention, context, common ground |
| Distributional | What does usage reveal about meaning? | Corpus co-occurrence structure |
| Embeddings | Can that structure become geometry? | Continuous vector spaces |
| Transformers | Can geometry become context-dependent? | Contextual states |
| LLMs | What converges when all of the above are optimized jointly? | Sequence prediction at scale |
Before Shannon: communication becomes an engineering object
Telegraphy, telephony, and radio created a practical need to quantify communication. Nyquist and Hartley studied the relationships among signaling rate, bandwidth, and the number of distinguishable messages — work Shannon explicitly identified as foundational. The decisive move was treating a source as selecting one message from a set of possibilities, which makes probability the natural language of the problem.
Shannon: information as uncertainty reduction
The 1948 paper A Mathematical Theory of Communication defined self-information and entropy, and with them conditional entropy, mutual information, redundancy, and channel capacity. Crucially, Shannon bracketed meaning deliberately — a methodological boundary, not a claim that meaning is unimportant.
An important correction to a common example: a bizarre sentence is not automatically high Shannon information. Its information depends on the probability assigned by a specified source model. Shannon surprise is model-relative, not a free-standing judgment of strangeness.
Semantics: the branch concerned with meaning
Frege’s 1892 distinction between sense and reference established that reference alone does not exhaust meaning — the morning star and the evening star pick out the same object but differ in sense. Tarski gave a rigorous account of truth for formal languages; Montague argued natural language could receive the same formal treatment. Compositionality — that the meaning of a complex expression is determined by its parts and their arrangement — became the organizing principle.
Pragmatics: meaning in context, intention, and action
Austin showed that utterances do things; Grice showed that what a speaker means routinely exceeds what a sentence says, through the cooperative structure of conversation. Deixis, presupposition, and common ground complete the picture.
Sentence meaning underdetermines speaker meaning; interpretation requires context, intention, and assumptions about cooperative interaction.
Distributional semantics: statistical traces of meaning
Harris and Firth proposed that a word is characterized by the company it keeps. The underlying logic:
If meaning influences how language is used, then patterns of use should carry recoverable evidence about meaning — even if those patterns do not exhaust meaning.
Word-context matrices with PMI weighting, and later dimensionality reduction, turned this into method. Levy and Goldberg later showed that skip-gram with negative sampling implicitly factorizes a shifted PMI matrix — the count-based and predictive traditions recover related structure.
Shannon information is not semantic content. Distributional semantics uses statistical structure as evidence about meaning. The semantic interpretation is an additional theoretical step.
Embeddings: distributional structure becomes geometry
An embedding is a learned mapping from a discrete object — a word, token, or sentence — into a continuous vector space, where geometric relationships carry information about similarity and relatedness. The 2003 neural probabilistic language model is the point at which embeddings and language modeling become a single objective rather than two stages; word2vec made it efficient at corpus scale.
Static embeddings are type-level representations: one vector per word, averaging across all its senses. That limitation is what contextual models exist to solve.
Transformers and contextual representation
Self-attention lets each token’s representation be computed as a weighted function of every other token, so the same word receives different states in different contexts. ELMo demonstrated contextuality before the Transformer using deep bidirectional LSTMs — evidence that contextuality is architecturally independent of attention. BERT’s masked bidirectional objective and autoregressive generative pretraining followed.
At this point “the embedding” stops referring to one object: input embeddings, positional encodings, per-layer hidden states, attention-head subspaces, and pooled outputs are all distinct things the word can name.
Large language models: the convergence point
An LLM directly optimizes sequence prediction. What it may learn indirectly — syntax, world knowledge, pragmatic regularity — is a separate question from what it optimizes, and several questions remain genuinely contested: understanding, grounding, intention, truth-conditional competence, and how supervision should be measured.
A model may reduce uncertainty without knowing truth, encode semantic regularities without complete grounding, reproduce pragmatic patterns without human intentions, and act competently while still imposing substantial verification and supervisory burden.
The bridges, stated precisely
Each transition carries something forward and leaves something behind. The third column is where most conceptual confusion originates.
| Transition | What carries across | What does not automatically carry across |
|---|---|---|
| Shannon → semantics | Probability over messages | Denotation, truth conditions |
| Semantics → pragmatics | Compositional structure | Speaker intention, context |
| Pragmatics → distributional | Meaning shapes usage | Intention, common ground |
| Distributional → embeddings | Co-occurrence structure | Sense distinctions, negation, scope |
| Embeddings → neural LMs | Learned representation | Grounding in a world |
| Neural LMs → Transformers | Prediction as objective | Causal understanding |
| Transformers → LLMs | Contextual composition | Epistemic reliability |
Six senses of the word “information”
Most cross-disciplinary confusion about AI and meaning traces to conflating these.
| Sense | Measures | Must not be confused with |
|---|---|---|
| Shannon information | Uncertainty reduction against a source model | Semantic content |
| Semantic information | Denotation and truth conditions | Statistical surprise |
| Pragmatic information | What a speaker conveys in context | Sentence meaning |
| Algorithmic (Kolmogorov) | Shortest description length | Shannon entropy |
| Fisher information | Sensitivity of a distribution to a parameter | Any of the above |
| Representational (ML) | Structure encoded in learned vectors | Information the model actually uses |
Distinctions worth holding onto
- Prediction versus meaning — optimizing one does not entail the other.
- Representation versus grounding — encoding structure is not connection to a world.
- Correlation versus compositionality — statistical association is not systematic structure.
- Sentence meaning versus speaker meaning — the gap pragmatics exists to describe.
- Encoded versus used information — a probe finding structure does not show the model uses it; that requires causal intervention.
- Fluency versus epistemic reliability — the most consequential distinction for evaluation.
Common misconceptions
- “Shannon information measures meaning.” It measures uncertainty reduction relative to a source model, and deliberately brackets meaning.
- “An embedding is a complete semantic representation.” It is a compressed record of distributional regularity, which is evidence about meaning rather than meaning itself.
- “Next-token prediction is merely autocomplete.” The objective is simple; what must be learned to satisfy it at scale is not.
- “Contextual embeddings solved grounding.” They solved sense-in-context. Grounding is a different problem.
Primary reading pathway
- Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal.
- Frege, G. (1892). Über Sinn und Bedeutung / On Sense and Reference.
- Tarski, A. (1933/1956). The Concept of Truth in Formalized Languages.
- Harris, Z. S. (1954). Distributional Structure.
- Firth, J. R. (1957). A Synopsis of Linguistic Theory, 1930–1955.
- Austin, J. L. (1962). How to Do Things with Words.
- Montague, R. (1970). Universal Grammar.
- Grice, H. P. (1975). Logic and Conversation.
- Deerwester, S. et al. (1990). Indexing by Latent Semantic Analysis.
- Bengio, Y. et al. (2003). A Neural Probabilistic Language Model.
- Mikolov, T. et al. (2013). Efficient Estimation of Word Representations in Vector Space.
- Levy, O. and Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization.
- Vaswani, A. et al. (2017). Attention Is All You Need.
- Peters, M. E. et al. (2018). Deep Contextualized Word Representations.
- Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers.