Building a model into an application
Calling a model from your code: the request (system role, messages, temperature, token limit), the response and its cost, structured output, tool calling, retrieval-augmented generation, the key kept on the server, prompt injection, timeouts and retries.
- 27 cards
- 6 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- The numbers
- On the diagram
- Scenarios
- Understand
A few cards from the deck
System prompt: what is it?
Show the answer
An instruction message placed at the head of the request, which sets the role, tone and rules the model must follow throughout the conversation.
It steers the model, it does not protect it: a user may try to make it repeat the prompt or get around it. No secret and no security rule the system depends on goes in it.
A web page calls the model's API directly from the browser, with the key written in its JavaScript. What is the problem?
- Any visitor can read the key and call the model at the owner's expense
- The browser is not allowed to send HTTPS requests to another address
- The model answers requests coming from a browser more slowly
- The key is erased from the code after its first successful call, which breaks the page
Show the answer
Any visitor can read the key and call the model at the owner's expense
A page's code is readable by everyone, as is the network traffic. An exposed key quickly ends up used by others, and the bill goes to the owner. The key stays on a server that acts as an intermediary.
A model's API key is kept …, read from an … variable; a website or mobile app that ships it hands it to anyone who opens its code.
Show the answer
A model's API key is kept on the server, read from an environment variable; a website or mobile app that ships it hands it to anyone who opens its code.
A hard-coded key ends up in the repository or in the shipped package. The browser or phone calls the application's server, which alone calls the model with the key, after identifying the user.
Structured output: what is it?
Show the answer
A response the model is constrained to produce in a format the code can read, usually a JSON object that matches a given schema.
Without a constraint, the model readily adds an introductory sentence around the JSON and parsing fails. The code still validates the response: a correct format does not guarantee correct content.