Language models
What a large language model does: split the text into tokens, predict the next token, append it and start again; context window, temperature, embeddings, the role of attention, pre-training then fine-tuning, and why it can state something false.
- 24 cards
- 5 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- Understand
A few cards from the deck
Token: what is it?
Show the answer
A chunk of text (a word, a word fragment, a punctuation mark) into which the text is cut before it enters the model; the model only sees these units, never the individual letters.
A common word often fits in a single token, a rare or long word takes several. A model's limits, prices and memory are counted in tokens, not in words.
An application extracts dates from contracts and wants answers as repeatable as possible from one call to the next. Which setting should it choose?
- A temperature close to 0
- A high temperature
- A larger context window
- More attention layers
Show the answer
A temperature close to 0
At a low temperature the most likely token is almost always chosen: answers repeat. A high temperature does the opposite. The window size and the number of layers do not touch the variety of the draw. Even at 0, perfect identity is not guaranteed.
The … mechanism lets each token weigh the other tokens of the context to build its representation; the architecture that stacks these layers is called the ….
Show the answer
The attention mechanism lets each token weigh the other tokens of the context to build its representation; the architecture that stacks these layers is called the transformer.
Its role: link a word to the ones that shed light on it, even far apart in the sentence (a pronoun and the noun it refers to). All positions are processed together, which makes training parallelisable.
Next-token prediction: what is it?
Show the answer
The task at the heart of a language model: from the beginning of a text, compute a probability for each of the units that may come next.
To write a text, the model draws one unit, appends it to the text, then starts again. The real continuation of an existing text serves as the expected answer: no human has to label the data.