Why Keywords Fail
in a Vector World

The Lexical Assumption Is Broken

For two decades, digital visibility depended on a simple assumption: if the right words appear on the right page, that page will be found. This is the lexical assumption. It powered SEO.

Large language models break this assumption completely.

LLMs do not scan pages for keywords. They encode documents into dense vector representations in high-dimensional space — typically 768 to 4,096 dimensions. Retrieval is determined by the geometric relationship between vectors, not by the presence of specific strings.

Cosine Similarity: The New Retrieval Metric

In embedding space, similarity between a query and a document is computed as the cosine of the angle between their respective vectors. This is a normalized dot product — a purely geometric measure.

The practical consequence is profound: two documents can contain completely different words and still be semantically close in embedding space. Conversely, two documents stuffed with identical keywords can be far apart if their semantic structures differ.

Keyword density optimizes for a metric that no longer governs retrieval. It is, mathematically speaking, optimizing for the wrong objective function.

Why Synonyms and Paraphrasing Collapse Lexical Strategy

Transformer architectures process language through attention mechanisms that capture contextual relationships between tokens. These architectures inherently handle synonyms, paraphrasing, ellipsis, and semantic compression.

This means a query phrased as "how to make a business visible to AI" will retrieve documents about "AI visibility engineering" even if those exact words never appear in the query.

Lexical matching cannot account for this. Vector similarity can.

Semantic Structure as the New Ranking Signal

If keywords are not the retrieval signal, what is?

Semantic structure. Embedding coherence. Entity clarity. Definitional stability.

Documents that present clear, structured, semantically consistent knowledge produce embedding vectors with low variance — they cluster tightly around a stable point in vector space. Documents with fragmented structure, mixed topics, or ambiguous terminology produce dispersed, high-variance embeddings.

Low-variance embeddings are more likely to fall within the retrieval region for relevant queries. High-variance embeddings scatter, reducing the probability of alignment.

This is the mathematical foundation of Answer Authority Engineering: optimizing the structure of knowledge so that embeddings are coherent, stable, and retrievable.

The Proprietary Layer

The principles above — cosine similarity, embedding coherence, semantic structure — are established mathematics. They are published science.

What 411bz has built is a proprietary engineering system that translates these mathematical principles into measurable, actionable infrastructure. Our scoring models, threshold calibrations, and optimization algorithms are proprietary.

We explain the physics. The engine stays in the vault.