From Shannon information to embeddings

Telegraphy, telephony, radio, and coding created a practical need to quantify communication. Harry Nyquist and Ralph Hartley studied the relationships among signaling rate, bandwidth, and the number of distinguishable messages — foundations Shannon explicitly credited.[1] The engineer’s questions — how many distinguishable signals, how fast, how noise corrupts, how much redundancy, how efficient an encoding — are not yet questions about what a message means. They concern selecting, encoding, transmitting, and reconstructing patterns.

In A Mathematical Theory of Communication, Shannon defined self-information as I(x) = −log₂ p(x) and entropy as average unpredictability, measured in bits. Core concepts — conditional entropy, mutual information, redundancy, channel capacity, noise — made communication rigorously measurable. The famous boundary: Shannon deliberately abstracted away semantics. This was a methodological boundary, not a denial of meaning; the same code carries false, profound, or nonsense statements alike. And surprise is model-relative — a bizarre sentence is not automatically high Shannon information.

Shannon’s experiments with English showed that language has substantial statistical structure, connecting prediction, compression, and linguistic regularity. But predictability is not understanding, and compressibility is not meaningfulness.

Early artificial intelligence asked whether intelligent behavior could be represented and produced through symbolic rules: theorem proving, formal languages, heuristic search. Its lasting contribution to the spine is the idea that meaning and inference can be treated as computational objects — a premise every later stage inherits, even the statistical ones that abandoned hand-built rules.

Semantics asks what kind of thing meaning is. Frege distinguished sense from reference — “morning star” and “evening star” both denote Venus, so reference alone does not exhaust meaning. Tarski made truth rigorous for formal languages via satisfaction relative to a model. Montague argued natural language is formally analyzable, and the principle of compositionality explains how finite speakers understand indefinitely many novel sentences. The tradition spans philosophical, model-theoretic, lexical, cognitive, inferential, and computational branches, with building blocks including denotation, truth conditions, possible worlds, intension and extension, quantification, scope, and anaphora.

Pragmatics studies what a speaker is doing in context. Austin and Searle decomposed utterances into locutionary content, illocutionary force, and perlocutionary effect. Grice showed how conversational implicature works: asked “Did every reviewer approve?”, the reply “Some of them did” licenses the inference that not all did. Deixis (I, you, here, today), presupposition (“Jordan stopped using the system” presupposes prior use), and common ground complete the picture. Each layer maps onto a live AI problem: reference resolution to coreference and grounding, intention to intent detection, illocutionary force to dialogue policy and safe action, discourse effect to memory and dialogue state.

Zellig Harris (Distributional Structure, 1954) and J. R. Firth anchored the distributional hypothesis: words occurring in similar linguistic contexts tend to have related meanings (doctor/nurse, cat/dog, encrypt/hash). Similarity is not identity — antonyms and topical associates also share contexts. Count-based word–context matrices operationalized the idea; since raw counts are dominated by function words, pointwise mutual information (PMI) and its positive variant became standard. Levy and Goldberg later showed that word2vec’s skip-gram implicitly factorizes a shifted PMI matrix — the predictive and count-based traditions recover related structure. Strengths: learns from natural data, scales, yields graded similarity. Limits: confuses association with synonymy or entailment, inherits corpus bias, struggles with negation, quantification, and scope, and captures no grounded perception or speaker intention.

Statistical NLP asked how linguistic structure and prediction could be learned from data: n-gram language models, hidden Markov models, probabilistic parsing, topic models, and information retrieval. Latent Semantic Analysis (Deerwester et al., 1990) showed that dimensionality reduction over co-occurrence counts recovers useful semantic structure — a direct precursor to learned embeddings.

An embedding is a learned mapping f(w) ∈ ℝ^d from a discrete object — a word, token, sentence, user, image, or graph node — into a vector space. Geometry contributes proximity, relational directions, clusters, and differentiable inputs; but the space depends on training data, context definition, architecture, objective, dimensionality, and tokenization. Word2vec (Mikolov et al., 2013) made predictive static embeddings practical via CBOW, skip-gram, and negative sampling. Static embeddings are type-level: bank gets one vector averaged over all its uses, so polysemy is blurred. The spaces capture similarity (car/automobile), relatedness (doctor/hospital), syntactic behavior, and domain structure — and also social stereotypes. “Close in the space” is not “same meaning.”

Factorizing p(x₁…xₙ) = ∏ p(xᵢ | x₁…xᵢ₋₁) links next-token prediction directly to Shannon’s conditional entropy and surprisal. Predicting well requires representing morphology, syntax, lexical compatibility, semantic roles, discourse, genre, factual associations, and pragmatic patterns. Bengio et al. (2003) fused embeddings and language modeling into a single objective; the lineage runs n-grams → feed-forward neural LMs → RNNs → LSTMs → attention → Transformers. Prediction need not be identical to understanding to produce useful representations — the dispute is over what those representations instantiate.

Attention Is All You Need (2017) introduced self-attention — Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V — letting each token’s representation depend on the whole sequence: bank resolves differently in “deposit money at the bank” and “sit on the river bank.” ELMo showed the shift from lookup to computation and that different layers encode different properties; BERT made deep bidirectional pretraining standard; autoregressive generative models made hidden states reusable for instruction following, dialogue, and coding. “The embedding” is no longer one object: input token embeddings, positional representations, hidden states, output embeddings, pooled embeddings, and retrieval embeddings are distinct.

Large language models are the convergence point of the spine: information-theoretic objectives (per-token cross-entropy), distributional evidence, dense representations, contextual composition, and post-training. What they may learn indirectly — grammar, facts, discourse, approximated pragmatics — is a separate question from what they optimize. Genuinely contested: whether prediction constitutes understanding, grounding, intention, truth-conditional competence versus fluency, and how human supervision should be measured.

It measures uncertainty reduction relative to a source model, and deliberately brackets meaning.

Surprise is model-relative. A bizarre sentence is only high-information against a particular source model, not absolutely.

Usage patterns are evidence about meaning, not an exhaustive account of it. Intention, grounding, and truth conditions are not recoverable from co-occurrence alone.

It is a compressed record of distributional regularity — evidence about meaning rather than meaning itself.

Contextual models compute a fresh representation for each occurrence; different layers encode different properties, so no single vector is “the” meaning.

The objective is simple; what must be learned to satisfy it at scale is not.

Fluency demonstrates learned regularity, not grounding, intention, or epistemic reliability. Contextual embeddings solved sense-in-context; grounding is a different problem.

Modern systems embed far more than words: image regions, audio segments, tool observations, task states, inferred goals, candidate actions, and intermediate computational states. The frontier question shifts from “How can meaning be represented?” to “How can a system maintain and revise a useful model of an evolving situation?”

Systems now compute after the problem arrives: generating alternatives, decomposing tasks, searching trajectories, invoking tools, running verifiers, and revising. Compute should be allocated by difficulty, uncertainty, expected value, risk, reversibility, and verification requirements.Next milestone: adaptive test-time compute.

Deployed AI wraps the model in retrieval, memory, tools, code execution, planning, verification, workflow state, access controls, human approval, and telemetry. Model-level evaluation misses retrieval errors, tool failures, stale memory, permission boundaries, and recovery — hence the shift toward system and trajectory evaluation.

Intent classification is a narrow subset of pragmatic competence. Fuller competence tracks shared commitments, unresolved ambiguities, competing interpretations, inferred intent with confidence, conversational branches, and repair history — conversation as a revisable graph of topics, claims, commitments, and corrections rather than a flat transcript.Next milestone: explicit common-ground management.

The window does not decide what to retain, how to represent it, when to retrieve it, whether it is still valid, or how to forget. The progression runs long context → retrieval memory → structured episodic memory → active memory management → continual consolidation, across working, episodic, semantic, procedural, institutional, and negative memory — each with its own risk (interference, privacy leakage, staleness, out-of-scope application, lost provenance, failure to suppress invalid knowledge).Next milestone: memory governance as an evaluation domain — retention correctness, deletion fidelity, provenance, conflict resolution, privacy boundaries.

Models accept text, images, speech, video, interface state, and sensor data, but grounding failures persist: shortcuts, misaligned reference, temporal integration errors. The progression is cross-modal alignment → multimodal reasoning → temporal grounding → unified event modeling → sensor-grounded action.Next milestone: the multimodal event model — who did what, to which object, when, where, under what constraints, with what observed result.

A world-model agent estimates how the environment responds, which states are reachable, likely failure modes, option effects, and reversibility — a loop of observe, plan, forecast, simulate, estimate risk, request authorization, execute, compare observed with predicted, and update. Embodied AI pushes this into the physical world, where semantics, pragmatics, perception, world modeling, and action become physically unified in proposed world-action models.Next milestone: prospective tool use.

Beyond chain-of-thought lies structured epistemic control: claims carrying their source, inference, uncertainty, counterevidence, verification status, reversibility, and responsible actor. In science and enterprise settings these evidence structures may become more valuable than fluent prose.

Probing, sparse autoencoders, causal interventions, circuit analysis, and activation steering aim at operational uses: detecting dangerous internal states, memorized private data, and conflicting goals; tracing evidence; predicting failure before action. Caveat: behavioral competence will likely advance faster than mechanistic understanding.

Enterprise deployments layer the foundation model with specialist services, retrieval and institutional memory, policy and authorization, workflow engines, validation, human escalation, and audit telemetry — an organizational control architecture. Benchmarks miss permissions, ambiguous ownership, changing data, exceptions, integration failures, recovery costs, and approval delays. The transition: from measuring whether an AI succeeds to measuring whether its path to success is supervisable, recoverable, and accountable.