What are embedding layers in deep learning and how are they different from one-hot encoding?

An embedding layer is a learnable lookup table that maps discrete tokens (words, categories, user IDs) to dense, low-dimensional real-valued vectors. It is mathematically a matrix E ∈ ℝ^{V×d} (vocabulary size × embedding dimension), and looking up token i simply retrieves row i — equivalent to multiplying a one-hot vector by E, but implemented as an O(1) table lookup rather than an O(V) matrix multiply.

The key advantage over one-hot encoding is that embeddings are learned — similar tokens (synonyms, related categories) naturally end up with similar embedding vectors because they appear in similar contexts during training. This gives embeddings semantic meaning and enables generalisation: the model can leverage the fact that 'Paris' and 'Berlin' are semantically similar even if 'Berlin' was rare in training data, because their embedding vectors will be nearby.

import torch
import torch.nn as nn

vocab_size  = 10000
embed_dim   = 128

embedding = nn.Embedding(
    num_embeddings=vocab_size,
    embedding_dim=embed_dim,
    padding_idx=0    # token 0 gets a fixed zero vector (PAD token)
)

# Input: integer token IDs
token_ids = torch.tensor([[1, 5, 23, 0], [42, 7, 0, 0]])  # (2, 4)
embedded  = embedding(token_ids)
print(embedded.shape)  # (2, 4, 128) — each token -> 128-dim vector

# Pre-trained embeddings (e.g. GloVe, Word2Vec)
pretrained = torch.randn(vocab_size, embed_dim)  # replace with real vectors
embedding.weight.data.copy_(pretrained)
# Freeze pretrained embeddings:
# embedding.weight.requires_grad = False

# In a text model:
class TextClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm  = nn.LSTM(embed_dim, 256, batch_first=True)
        self.fc    = nn.Linear(256, 5)
    def forward(self, x):
        e = self.embed(x)          # (B, L, 128)
        _, (h, _) = self.lstm(e)
        return self.fc(h[-1])