Tools & Data
What Are LLM APIs?
The programmatic interfaces to large language models, and the parameters that decide whether output is reproducible.
Definition
An LLM API is an HTTP interface for sending prompts to a large language model and receiving generated output. The main providers are Anthropic, OpenAI and Google. Requests are billed by token, counting both the input sent and the output produced.
Concepts common to all of them
- Tokens: the units text is split into, which determine both cost and context limits
- Context window: the maximum tokens a single request may contain
- System prompt: instructions that frame the whole exchange, separate from the user turn
- Temperature: how much randomness is allowed in sampling, where zero is closest to deterministic
- Tool or function calling: letting the model request a structured call your code executes
- Streaming: receiving the response incrementally rather than waiting for completion
Making output reproducible enough to test
Generation is sampled, so identical inputs can produce different outputs. For pipelines that need to be testable, set temperature low, pin the model version rather than an alias that moves, request structured output where the provider supports it, and validate the response shape before using it.
Cost control
Input tokens usually dominate in retrieval heavy applications, because the retrieved passages are large relative to the question. Prompt caching, offered by the major providers, reduces the cost of repeatedly sending the same prefix. Trimming retrieved context is normally a bigger saving than shortening the instructions.
Tokens are the smallest individual units of a language model, and can correspond to words, subwords, characters, or even bytes.