AI / LLM Basics Interview Questions
What is an Embedding in the context of LLMs?
An embedding is a list of numbers, a vector, that represents a token's meaning in a way a neural network can do math with.
- Words with similar meanings end up with embeddings that are mathematically close to each other
- Generated by an embedding layer that's learned during training, not manually assigned
- Used as the very first step inside the model, converting each input token into its vector form before any further processing happens
Beyond just being an internal step inside an LLM, embeddings are also used directly for tasks like semantic search and clustering, where comparing the "closeness" of two pieces of text's meaning is the whole point.
What does an embedding represent a token as?
How are embeddings assigned to tokens?
More Related questions...
