Embeddings

Vector Representations of Language

Word embeddings are a revolutionary technique in natural language processing (NLP) that transforms words into numbers that computers can understand while preserving the words’ meanings. At their core, embeddings represent words as vectors – lists of numbers – in a way that captures the relationships between words. Words with similar meanings end up with similar number patterns, allowing computers to understand language in a more human-like way.

Basic Concept: From Words to Numbers

The Challenge of Teaching Computers Language

Computers can only process numbers, not words. Traditional approaches to converting words to numbers, like assigning each word a unique ID (e.g., “cat” = 1, “dog” = 2), fail to capture any meaningful relationships between words. For instance, in this system, the computer has no way of knowing that “cat” and “dog” are more related to each other than “cat” and “telescope.”

The Embedding Solution

Word embeddings solve this problem by representing each word as a vector (a list of numbers) in a way that preserves meaning. For example, a simple 3-dimensional embedding might represent words like this:

cat = [0.2, 0.5, -0.1]
dog = [0.1, 0.4, -0.2]
pet = [0.3, 0.5, -0.1]
telescope = [-0.5, 0.1, 0.8]

In this simplified example, notice how the numbers for “cat,” “dog,” and “pet” are similar to each other but quite different from “telescope.” This reflects their real-world relationships – cats, dogs, and pets are related concepts, while a telescope is quite different.

How Embeddings Work

The Vector Space

Imagine a vast multi-dimensional space where every word in a language has its own point or location. This is called the “embedding space.” While our example above used just 3 dimensions for simplicity, real-world embeddings typically use anywhere from 50 to 300 dimensions to capture the subtle nuances of meaning.

Key Properties:

1. Similar Words Cluster Together

• Words with similar meanings appear close to each other in this space
• Example: “king,” “queen,” “prince,” and “princess” would form a cluster
• Another cluster might include “dog,” “puppy,” “cat,” and “kitten”

1. Meaningful Directions

• The directions between words often capture meaningful relationships
• Famous example: vector(“king”) – vector(“man”) + vector(“woman”) ≈ vector(“queen”)
• This shows that embeddings can capture analogical relationships

1. Distance Equals Similarity

• The closer two words are in the embedding space, the more related they are
• This is often measured using “cosine similarity” – the angle between their vectors
• Example: “happy” and “joyful” would have a high cosine similarity

How Embeddings Are Created

Embeddings are learned from large amounts of text using neural networks. The process works on a simple principle: words that appear in similar contexts should have similar embeddings.

Training Process:

1. Context WindowContext Window – The span of text that a language model ca... learn this...

• The system looks at chunks of text, perhaps 5-10 words at a time
• Example: In “The cat sat on the mat,” when looking at “sat,” the context words are “the,” “cat,” “on,” “the,” “mat”

1. Prediction Task

• The neural network tries to predict a word given its context, or vice versa
• It adjusts the word vectors to make better predictions
• Over time, words used in similar contexts get similar vectors

1. Dimensionality

• Each dimension potentially captures a different aspect of meaning
• Early dimensions might capture obvious things like “is this a living thing?”
• Later dimensions capture more subtle distinctions

Real-World Applications

1. Search Engines

When you search for “cars for sale,” a search engine using embeddings can also match related phrases like “automobiles available” or “vehicles to purchase” because their embeddings are similar.

Example of how it works:

1. Your search query “cars for sale” is converted to a vector
2. The system finds documents whose word vectors are similar
3. This allows for semantic search rather than just keyword matching

2. Language Translation

Embeddings help machine translation by creating a language-independent space of meaning:

English: "cat" = [0.2, 0.5, -0.1]
Spanish: "gato" = [0.19, 0.48, -0.12]
French: "chat" = [0.21, 0.51, -0.09]

These similar vectors help the system know these words mean the same thing.

3. Autocomplete and Text Prediction

When you’re typing, systems use embeddings to suggest likely next words based on similarity in the embedding space.

Technical Implementation

Common Embedding Models

1. Word2Vec (2013)

• The first widely successful embedding model
• Uses two architectures:
  • Skip-gram: Predicts context words from a target word
  • CBOW (Continuous Bag of Words): Predicts a target word from context words

1. GloVe (Global Vectors)

• Focuses on global word co-occurrence statistics
• Often captures slightly different relationships than Word2Vec

1. FastText

• Breaks words into subwords
• Can handle misspellings and unknown words
• Example: “playing” is represented using vectors for “play,” “ing,” “aying,” etc.

Measuring Similarity

The most common way to measure similarity between embeddings is cosine similarity:

def cosine_similarity(v1, v2):
    dot_product = sum(a * b for a, b in zip(v1, v2))
    magnitude1 = math.sqrt(sum(a * a for a in v1))
    magnitude2 = math.sqrt(sum(b * b for b in v2))
    return dot_product / (magnitude1 * magnitude2)

Limitations and Challenges

1. Ambiguity

Single embeddings struggle with words that have multiple meanings:

• “bank” (financial institution vs. river bank)
• “spring” (season vs. coiled metal vs. to jump)

2. Context Sensitivity

Traditional embeddings give each word a single vector, regardless of context. Modern systems like BERT create context-dependent embeddings, but they’re more complex.

3. Bias

Embeddings can inherit biases present in their training data:

• Gender stereotypes
• Cultural biases
• Historical biases

Latest Developments

Contextual Embeddings

Newer models like BERT and GPT create different embeddings for the same word based on context:

• “The bank approved my loan” → financial meaning
• “The bank of the river” → geographical meaning

Multimodal Embeddings

Recent research combines text embeddings with:

• Image embeddings
• Audio embeddings
• Video embeddings

Conclusion

Word embeddings represent a fundamental breakthrough in how computers process language. By converting words into vectors in a way that preserves meaning, they enable a wide range of modern NLP applications. While they have limitations, ongoing research continues to improve their capabilities and address their shortcomings.

See Also

• Neural Networks
• Natural Language Processing
• Vector Space Models
• Semantic Analysis