Chapter

Vector Search: Similarity Is Not Truth

Part G · From Records to Insight|3,807 words|about 17 min read|Volume G

45.0 What this chapter gives you#

Vector search represents items as lists of numbers and retrieves items that are close under a chosen mathematical rule. It can find useful similarities even when the exact query words do not appear in the result.
You will calculate distances, compare exact and approximate retrieval, inspect index trade-offs and test permission filters. The central boundary is simple: closeness in a representation is not proof of factual correctness, identity or authorization.
The companion exercises use small explicit vectors and direct Python arithmetic. They do not train an embedding model, implement HNSW or run Faiss, and they do not claim production semantic-search quality.

45.1 Representing items as vectors#

45.1.1 PLAIN — in simple words#

A vector is an ordered list of numbers. For search, a procedure turns each document, image or other item into such a list so similar items may occupy nearby positions.
The numbers gain meaning from the procedure that created them. Two lists of the same length are not automatically comparable if they came from different models or preprocessing rules.
A vector is a representation, not the original item. Keep a reliable link to the source and its version so a result can be inspected rather than trusted because a number looks impressive.

45.1.2 PLAIN — a picture in your head#

Mira makes a map where cards about similar topics tend to sit near one another. A new question is placed on the same map, and she inspects nearby cards.
The map helps find candidates, but a nearby card might be outdated, misleading or about the wrong sense of a word.
Where the comparison breaks: learned vectors usually have many dimensions that are not individually labelled like streets. A coordinate does not necessarily correspond to a human-readable topic, and the geometry depends on the training objective.

45.1.3 PLAIN — a worked example#

Begin with an invented two-dimensional representation rather than a trained model. Let the query be Q = (1, 0), and items A = (1, 0), B = (0.8, 0.6), C = (0, 1), D = (-1, 0) and E = (2, 0).
These coordinates are teaching data. No assertion is made that a real document about pens naturally receives any of these numbers.
A search algorithm can rank the items once a metric is chosen. It cannot decide whether A’s underlying text is correct merely because A’s vector equals the query vector.
If a different encoder maps the same source into another coordinate system, combining its output with the old index without compatibility checks can make the ranking meaningless.

45.1.4 PLAIN — what is really happening inside#

An embedding pipeline extracts and preprocesses content, runs an encoder and stores the resulting vector with source identity and metadata.
Query encoding must use a compatible procedure. Some systems use different query and document encoders trained to work together; compatibility is therefore more specific than simply using identical software names.
Content updates, chunking changes or encoder revisions can require re-embedding. An index containing mixed generations needs an explicit compatibility policy and evaluation rather than accidental coexistence.

45.1.5 TECHNICAL — the engineer’s version#

Store encoder identity/version, vector dimension, numerical type, normalization policy, source revision and chunk boundaries. These attributes are part of the retrieval contract.
Validate dimensions and finite numeric values before distance computation. NaN, infinity or a zero vector can invalidate a selected comparison, especially cosine normalization.
Faiss exposes several index and metric choices rather than one universal meaning of vector proximity. Its documented metric conventions must be matched to the representation and task. [S176] [S177]

45.1.6 WORDS — remember these#

Embedding: an item’s numerical representation — an ordered vector produced by a defined encoder and preprocessing pipeline. Embedding space: the coordinate system used for comparison — a representation whose geometry depends on its model and training objective. Representation compatibility: vectors can be compared meaningfully — agreement between encoder, dimension, normalization and metric contracts.

45.2 Distance and similarity#

45.2.1 PLAIN — in simple words#

Euclidean distance measures straight-line separation. Inner product multiplies corresponding coordinates and adds the results. Cosine similarity compares direction after accounting for vector length.
These rules can rank the same items differently. Choosing one is part of defining what “similar” means for the application.
A smaller distance is usually better under a distance rule, while a larger similarity is usually better under a similarity rule. Always check the actual API’s returned quantity and direction.

45.2.2 PLAIN — a picture in your head#

Imagine arrows from the centre of a map. Two arrows can point in the same direction but have different lengths.
Cosine treats direction as important and ignores that length difference. Euclidean distance still sees the separation between the arrow tips.
Where the comparison breaks: the choice of geometry is not a universal model of human meaning. A trained representation may rely on length, direction or another scoring rule, so changing the metric can damage retrieval.

45.2.3 PLAIN — a worked example#

For Q = (1, 0) and B = (0.8, 0.6), squared Euclidean distance is (1 - 0.8)^2 + (0 - 0.6)^2 = 0.4. The distance itself is the square root, about 0.632.
The squared distances from Q to A, B, E, C and D are respectively 0, 0.4, 1, 2 and 4. Exact Euclidean ranking therefore places them in that order, with a defined tie-break if needed.
Inner products with Q are A = 1, B = 0.8, C = 0, D = -1 and E = 2. Maximum-inner-product search prefers E to A in this example.
Cosine similarities are A = 1, B = 0.8, C = 0, D = -1 and E = 1. A and E point in the same direction despite their different lengths. A zero vector has no defined cosine direction and is rejected by our model.

45.2.4 PLAIN — what is really happening inside#

Distance evaluation performs arithmetic across dimensions. Exact search repeats this comparison for every eligible stored vector unless another exact structure can safely eliminate candidates.
Squared Euclidean distance preserves Euclidean ordering because the square root is monotonic for non-negative inputs. It does not preserve the numerical unit of distance itself.
Normalizing vectors changes their length to one. For unit vectors, squared Euclidean distance equals 2 - 2 × inner_product, connecting cosine and Euclidean rankings under those assumptions.

45.2.5 TECHNICAL — the engineer’s version#

Faiss reports squared L2 distance for its L2 metric. Its inner-product search is not automatically cosine similarity; cosine comparison requires suitable normalization of both queries and stored vectors. [S176]
The local functions check equal dimensions, finite values and nonzero norms where required. They use deterministic ID tie-breaking so repeated results can be compared without interpreting tie order as evidence of relevance.
A score threshold must be evaluated for the chosen encoder, metric and task. A value such as 0.8 has no universal meaning of “80% factually correct” or “the same person.”

45.2.6 WORDS — remember these#

Euclidean distance: straight-line separation — the square root of summed squared coordinate differences. Inner product: coordinate products added together — a similarity score affected by stored-vector magnitudes as well as direction. Cosine similarity: directional agreement — inner product divided by the product of nonzero vector norms.

45.3 Exact and approximate search#

45.3.1 PLAIN — in simple words#

Exact nearest-neighbour search returns the closest eligible items under the declared metric. It says nothing about whether that metric captures the reader’s real need perfectly.
Approximate search saves work by examining a selected subset or a compressed representation. It can miss a mathematically nearer item.
Approximation is useful when its measured cost-quality trade-off fits the application. Calling it approximate should lead to evaluation, not to assuming that errors are either impossible or harmless.

45.3.2 PLAIN — a picture in your head#

Mira can measure the distance from a question to every card on the map, or search only promising neighbourhoods first.
The neighbourhood shortcut is faster when it avoids much work, but the closest card might be just outside the searched area.
Where the comparison breaks: different indexes choose candidates using graphs, partitions, quantized codes or other structures. The map picture does not establish the recall or runtime of any particular algorithm.

45.3.3 PLAIN — a worked example#

In a separate fixture, Q = (0, 0), A = (0.1, 0), B = (0.2, 0), C = (5, 5) and D = (5.1, 5). The exact top two under Euclidean distance are A and B.
Suppose a deliberately limited candidate stage returns only A and C. Reranking those candidates exactly still returns A and C; it cannot recover B because B was never considered.
The overlap with the exact top-two set is one item out of two, so neighbour recall at two is 0.5. This is approximation recall against the metric’s exact answer, not human relevance recall.
Expanding the candidate set to A, B and C recovers the exact top two in this tiny example. It does not prove that a fixed candidate budget will achieve the same recall on a different collection.

45.3.4 PLAIN — what is really happening inside#

Candidate generation reduces the number of full comparisons. Approximate indexes attempt to locate useful regions or neighbours without exhaustively scanning all vectors.
Reranking can improve ordering within the candidate set, especially when the first stage used compressed scores. Its maximum possible recovery remains limited by candidate coverage.
An exact baseline is valuable for measuring approximation loss on representative queries. Human-labelled relevance is another baseline, because exact geometric neighbours can still be poor answers.

45.3.5 TECHNICAL — the engineer’s version#

Faiss distinguishes exhaustive flat indexes from non-exhaustive structures such as IVF and HNSW. Index parameters affect the balance between search work and recovered neighbours. [S177]
Define recall@k against a fixed exact result set, including tie handling and eligible filters. A candidate stage and a reranker should be measured separately when diagnosing omissions.
The companion approximation example intentionally restricts a candidate list. It demonstrates the coverage limit, not the internal behaviour or performance of an ANN library.

45.3.6 WORDS — remember these#

Nearest neighbour: an item closest under a metric — a geometric result whose usefulness depends on the representation. Approximate nearest-neighbour search: a shortcut to likely close items — retrieval that may miss exact neighbours to reduce cost. Candidate recall: how much required material reaches the later stage — coverage that limits what reranking can ultimately recover.

45.4 Index trade-offs#

45.4.1 PLAIN — in simple words#

A vector index organizes stored vectors so a query can search more selectively. Building and maintaining that organization costs memory, computation and update work.
Some designs search connected neighbours in a graph. Others first choose coarse groups and search selected groups. Compression can reduce stored vector size while introducing approximation.
There is no single best setting independent of data, queries and operating requirements. A setting that works well on one demonstration may behave differently after growth or distribution changes.

45.4.2 PLAIN — a picture in your head#

Mira can connect each card to nearby cards and follow those links, or divide the map into districts and inspect selected districts.
More links or more inspected districts can improve the chance of finding the right card, but consume additional space or query work.
Where the comparison breaks: graph construction and partition training have precise algorithms and implementation details. A human map does not prove search complexity, deletion support or behaviour under concurrent updates.

45.4.3 PLAIN — a worked example#

One million vectors with 768 coordinates stored as 32-bit floats require 1,000,000 × 768 × 4 = 3,072,000,000 raw vector bytes, or 3.072 decimal GB.
That excludes identifiers, metadata, graph links, allocator overhead and replicated copies. Quoting only the raw vector size as the service’s total memory requirement would be incomplete.
A hypothetical product-quantized representation using 96 one-byte subvector codes has 96 code bytes per vector instead of 3,072 float bytes. Codebooks, IDs and optional original vectors still add storage, and distance estimates may become approximate.
A coarse-group index that probes 10 groups instead of 2 usually performs more candidate work. The actual improvement in recall must be measured; the group count alone does not guarantee a fixed accuracy level.

45.4.4 PLAIN — what is really happening inside#

HNSW builds a hierarchy of neighbour graphs and navigates from sparse upper layers toward a more detailed lower layer. Its search effort is controlled by algorithm and implementation parameters.
Inverted-file vector indexes assign vectors to coarse regions. A query selects regions to probe and evaluates candidates within them, potentially using compressed codes before exact reranking.
Updates, deletions and model changes require maintenance policies. Some index forms rebuild, mark deletions or support limited mutation patterns; the application must not assume every library supports the same lifecycle.

45.4.5 TECHNICAL — the engineer’s version#

The original HNSW research describes hierarchical graph navigation for approximate neighbour retrieval. Its empirical results and design do not constitute a universal latency or recall guarantee for every modern implementation. [S178]
Faiss documents distinct storage formulas and supported operations for its index classes. Use the selected class’s actual behaviour and measured overhead when planning capacity or update handling. [S177]
Benchmark build time, index memory, query latency distributions, recall, filtered retrieval and update/deletion behaviour. Report the exact encoder, metric, data distribution, parameters and hardware before comparing results.

45.4.6 WORDS — remember these#

HNSW: hierarchical neighbour-graph search — an approximate indexing approach navigating graphs at several levels of detail. IVF index: search through selected coarse groups — an inverted-file organization assigning vectors to regions before candidate evaluation. Product quantization: compact codes for vector parts — a representation replacing subvectors with codebook references for storage and distance estimation.

45.5 Filtering and evaluation#

45.5.1 PLAIN — in simple words#

Search may need to consider only a permitted tenant, language, document type or time range. Those filters define the eligible population before usefulness is judged.
Taking the global top few and filtering afterward can return too few eligible results even when good permitted results exist nearby.
Permission filtering is not merely a quality adjustment. Protected content, scores, titles and snippets must not leak through intermediate or fallback paths.

45.5.2 PLAIN — a picture in your head#

Mira has separate cabinets for two clients. A client asks for the nearest two cards in their own cabinet.
Dev first chooses the nearest two from both cabinets, then removes the other client’s cards. He may return nothing while two suitable cards remain in the correct cabinet.
Where the comparison breaks: actual indexes support filters in different ways, sometimes before search, during graph traversal or after candidate generation. Correct access enforcement and recall under filtering both require verification.

45.5.3 PLAIN — a worked example#

Let Q = (0, 0). Documents V1 and V2 belong to tenant A and lie at distances 0.1 and 0.2. V3 and V4 belong to tenant B and lie at distances 0.3 and 0.4.
A tenant-B request for two results should consider V3 and V4. Searching the global top two first yields V1 and V2; filtering afterward leaves zero results.
Searching the eligible tenant-B set exactly returns V3 and V4. Overfetching more global candidates can help this example, but a fixed overfetch count does not guarantee sufficient eligible results for every distribution.
The tenant scope in the lab is supplied by an explicit trusted test context. It is not accepted as arbitrary proof from a caller, and the exercise does not implement a production login or authorization provider.

45.5.4 PLAIN — what is really happening inside#

Metadata filters interact with index traversal and candidate limits. The eligible population can be much smaller or differently distributed than the overall corpus.
Evaluation should measure recall after the actual filters, not only on unfiltered queries. Rare tenants or restrictive time ranges can expose failures hidden by aggregate results.
Empty-result and fallback behaviour matters. Relaxing a permission filter to produce an answer is not an acceptable relevance improvement; changing an optional user preference requires a visible policy.

45.5.5 TECHNICAL — the engineer’s version#

Distinguish mandatory authorization predicates from optional relevance filters. The former must remain enforced through candidate retrieval, reranking, caching and display.
Report exact-neighbour recall and human relevance metrics separately. The first measures approximation relative to geometry; the second measures whether the retrieved source helps with the labelled task. [S181]
Test filter-before-limit behaviour, cross-tenant requests, stale metadata, deleted sources and empty eligible sets. Passing vector-distance tests alone does not establish these boundaries.

45.5.6 WORDS — remember these#

Eligible set: items permitted by the request’s complete conditions — the population within which retrieval should answer the question. Post-filtering: removing candidates after retrieval — an approach that can reduce returned coverage when the candidate budget was spent on ineligible items. Overfetching: retrieving extra candidates before later selection — a mitigation that must be evaluated rather than assumed to guarantee filtered recall.

45.6 Retrieval limits#

45.6.1 PLAIN — in simple words#

A nearby document may be wrong, obsolete or about a subtly different issue. Similarity is a route to evidence, not a replacement for reading that evidence.
When a retrieved passage is supplied to a text-generating system, the generated answer adds another stage that can misinterpret, omit or invent information.
Good retrieval should preserve source identity, version, context and permission. A plausible answer with no trustworthy connection to its sources is difficult to verify.

45.6.2 PLAIN — a picture in your head#

Mira finds a card near the question on her topic map. She opens the full source before using it to make a decision.
A helper who writes a fluent summary can still misunderstand the card. The map and the summary each need their own checks.
Where the comparison breaks: retrieved documents can contain untrusted instructions as well as facts. A system using external text must not grant that text authority to change access rules or execute operations.

45.6.3 PLAIN — a worked example#

Suppose a passage about backup creation is geometrically close to a question about restore verification. It may discuss related vocabulary but omit the steps needed to establish that a restore actually works.
A retrieval score of 0.92 does not fill that gap. The answer must distinguish what the passage supports from what still needs another source or an experiment.
If the retrieved document belongs to an older software version, its commands may not apply to the current environment. Source version and consultation date are relevant evidence, not decorative metadata.
A system can correctly retrieve all three passages selected by its metric and still answer the wrong question. Evaluate retrieval, source interpretation and final factual support as separate stages.

45.6.4 PLAIN — what is really happening inside#

Retrieval pipelines often combine lexical and vector candidates, merge them and rerank. Each stage can improve one failure mode while introducing another, such as duplicate passages crowding out diverse sources.
A source-grounded answer must link claims to actual supporting text and retain relevant limitations. A citation to a merely related passage is not adequate support.
Updating or deleting a source requires corresponding treatment of embeddings, indexes, caches and retained outputs. Removing the original file alone does not establish that every derived copy has disappeared.

45.6.5 TECHNICAL — the engineer’s version#

Separate four contracts: encoder semantics, approximate retrieval quality, access eligibility and answer support. Success at one layer does not establish the others.
Treat retrieved text as data from its actual trust level. It must not override system instructions, authorize external actions or bypass tenant restrictions merely because it was returned by an index.
The laboratory proves selected geometry, candidate-coverage and filtering examples under declared finite inputs. It makes no claim that vector proximity establishes identity, truth, safety or universal semantic understanding.

45.6.6 WORDS — remember these#

Hybrid retrieval: combining different candidate methods — often lexical and vector search followed by merging or reranking. Grounded answer: a response supported by inspected evidence — claims linked to sources that actually establish them, with limitations retained. Retrieval boundary: what finding a source does and does not prove — the distinction between candidate similarity and verified, authorized factual support.

45.97 Practice and worked answers#

Question: Can vectors from two unrelated encoders be compared merely because both have 768 coordinates? Answer: No. Dimension equality does not establish representation compatibility.
Question: What is squared distance from (1, 0) to (0.8, 0.6)? Answer: 0.4. The Euclidean distance is approximately 0.632.
Question: Why do A = (1, 0) and E = (2, 0) tie under cosine with Q = (1, 0)? Answer: They point in the same direction; cosine removes their length difference.
Question: Why reject a zero vector for cosine in the lab? Answer: Its norm is zero, so the normalization denominator and directional comparison are undefined.
Question: Exact top two are A/B, but candidates contain A/C. What is neighbour recall at two? Answer: One recovered exact neighbour divided by two, or 0.5.
Question: What does the raw 3.072 GB estimate omit? Answer: IDs, metadata, index structures, memory overhead, replicas and any additional stored representations.
Question: Why can global top-two retrieval followed by tenant filtering return nothing? Answer: The candidate budget may be filled entirely by ineligible documents before filtering.
Question: Does similarity 0.92 mean a passage is 92% true? Answer: No. It is a metric-dependent score, not a calibrated factual-correctness probability.

45.98 Common wrong ideas#

Wrong: A vector contains an item’s complete meaning. Right: It is a representation learned or defined for particular objectives.
Wrong: Equal dimensions make models interchangeable. Right: Encoder and preprocessing compatibility must be established.
Wrong: Inner product is always cosine. Right: Normalization and nonzero norms matter.
Wrong: Exact vector search guarantees a relevant answer. Right: It guarantees the selected geometric result, not human usefulness.
Wrong: Reranking can repair every candidate omission. Right: Missing candidates remain unavailable.
Wrong: An index’s raw-vector size is total service memory. Right: Index and operating overhead can be substantial.
Wrong: Permission filters may be relaxed to improve recall. Right: Authorization defines the allowed population.
Wrong: A high-scoring citation proves the generated answer. Right: The cited text must actually support each claim.

45.99 Chapter summary in 20 lines#

Vectors are ordered numerical representations.
Their meaning depends on the encoder and preprocessing.
Preserve source identity and representation version.
Validate dimensions and finite values.
Euclidean distance measures separation.
Squared distance preserves order but changes the returned quantity.
Inner product depends on vector magnitudes.
Cosine compares nonzero vector directions.
Exact search is exact only under its declared metric.
Approximate retrieval can omit closer items.
Reranking cannot recover absent candidates.
Measure geometric recall separately from human relevance.
Graphs and coarse groups trade work for candidate coverage.
Compression saves space with additional accuracy and decoding trade-offs.
Capacity includes more than raw vector bytes.
Filters define the eligible retrieval population.
Post-filtering after a small global limit can lose useful results.
Retrieved content remains subject to access and trust boundaries.
Similarity is not truth, identity or authorization.
Inspect the source and verify what the answer actually supports.

Return to contents