Embeddings and Similarity

Meaning as coordinates

Neural networks work on numbers, so each token is mapped to a list of numbers called an embedding (hundreds to thousands of values). During training, tokens used in similar contexts end up with similar vectors. Cosine similarity (the angle between two vectors) measures closeness: near 1 means very similar direction, near 0 unrelated. The same idea powers semantic search: embed documents and the query, then return the nearest vectors. The numbers below are hand-made to show the idea; real embeddings are learned.

Numbers in, probabilities out

Embeddings turn tokens into vectors; softmax turns scores into probabilities; sampling picks a token.

Cosine similarity, run

I ran this plain-Python (standard library only) example. "king" and "queen" point in almost the same direction (0.994); "king" and "apple" do not (0.303).

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

emb = {
    "king":   [0.9, 0.8, 0.1],
    "queen":  [0.9, 0.7, 0.2],
    "apple":  [0.1, 0.2, 0.9],
}
print("king~queen", round(cosine(emb["king"], emb["queen"]), 3))
print("king~apple", round(cosine(emb["king"], emb["apple"]), 3))

Output:

king~queen 0.994
king~apple 0.303

Embeddings are model-specific

Vectors from different embedding models are not interchangeable. If you switch models, re-embed all your documents.

Quick check: What does a cosine similarity near 1 indicate?

• The vectors are unrelated
• The vectors point in nearly the same direction
• The text is long
• The model is large