The variety of available LLM models is valuable. More specialized options, more price points, more deployment flexibility. But that same variety is what makes selection difficult. Open-weight models like Llama and DeepSeek sit alongside commercial APIs from OpenAI, Anthropic, and Google. Domain-specific models trained for legal, medical, or financial work add another layer. Each comes with its own strengths, limitations, and cost structure.
The temptation when facing this many options is to reach for whatever is trending or to default to the model your engineering team already knows. Neither leads to a selection decision you can defend when requirements tighten, scale increases, or security questions surface later in the process.
7 Factors to Consider when Evaluating LLM Models
The 7 factors below give you a way through, sequenced so that the constraints most likely to eliminate a model surface before you’ve spent time evaluating it on everything else.
Evaluating LLM Output Quality
Quality is where evaluation starts in most cases. Before you test anything, define what good output looks like for your specific use case, concretely. What does a correct response look like? What does an acceptable one look like? Where is the line?
Quality is not uniform across a model’s capabilities. A model that performs well on straightforward requests may degrade on edge cases, longer inputs, or tasks that require multi-step reasoning. Your test inputs should reflect what users will actually send: representative samples across the full range of your use case, including the hard ones, not generic prompts.
Output reliability is a distinct dimension. Quality is about whether the model produces good answers. Output reliability is about whether the model follows instructions consistently: whether it returns valid JSON when asked, respects format constraints, and adheres to system prompt instructions across sessions. For teams integrating LLMs into workflows where output gets parsed or acted on programmatically, inconsistent following or instructions can sink a model that scores well on everything else. Define your reliability requirements as precisely as your quality requirements before you start testing.
When evaluating models, the natural starting point is public benchmarks and leaderboard scores, which rank models across standardized tests of reasoning, coding, language understanding, and other capabilities. They are useful for a first pass at narrowing the field, but they are not a substitute for evaluating a model against your specific domain. A model that ranks highly on a general benchmark may underperform on your particular use case, and vice versa.
Start with the most capable model available. This gives you a ceiling: a clear picture of what the output can look like at its best. From there, you can make informed tradeoffs against latency, cost, and the other factors that follow.
Context Window: A Hard Limit, Not a Tradeoff
The context window is the maximum amount of text a model can process in a single interaction. That includes your system prompt, the user’s input, any documents or data you provide, and the conversation history. When the total exceeds the model’s limit, the model cannot process the request.
This is a hard limit and not a performance tradeoff. A model does not degrade gracefully when it runs out of context. It fails. For use cases involving long documents, extended multi-turn conversations, or retrieval-augmented generation where chunks of external content are injected into the prompt, the context window is an architectural constraint that needs to be verified before you commit to a model.
Context window size varies significantly across models. Verify that the model you are evaluating can accommodate your expected payload at the high end, not the average.
LLM Latency: The Three Metrics That Matter
Latency is not a single number. First, identify whether you are steaming or delivering a full response, and evaluate your model using these 3 metrics:
Time to first token (TTFT) is how long it takes the model to begin responding after receiving a prompt. For streaming applications, where the response appears word by word as it generates, this is what determines how quickly users see anything at all. A high TTFT means users are staring at a blank screen, which affects perception of responsiveness even if the total response time is acceptable.
Inter-token latency is the time between each generated token. In a streaming context, this is what determines how smoothly the response flows to the user. When inter-token latency is high, the response arrives in uneven bursts rather than a steady stream, noticeable even when TTFT is fast. When you are waiting on a full response rather than streaming, inter-token latency matters only as a component of total response time.
Total response time is what matters when your application waits for the full response before displaying anything. It can be estimated as TTFT plus the number of output tokens multiplied by inter-token latency. For batch processing or background tasks, this is the number your users actually feel.
Reasoning models, those that work through a problem before generating output, tend to have significantly higher TTFT because processing happens before the first token appears. That delay can be worth it when accuracy on complex tasks is the priority, but it needs to be a deliberate decision, not a surprise in production.
Scaling Your LLM: Rate Limits and Infrastructure
Scale introduces two constraints: requests per minute (RPM) and tokens per minute (TPM).
- RPM caps how many API calls your application can make in a given minute regardless of their size.
- TPM caps the total volume of tokens processed in that same window, input and output combined.
In practice, TPM tends to be the binding constraint before RPM. A single request can consume thousands of tokens, so token volume accumulates far faster than request count.
Plan for peak demand, not average demand. The gap between the two can be significant depending on your adoption pattern, and you size your solution for the peak. When you exceed your limits, the consequences are immediate because requests fail and responses degrade.
Before committing to a model or provider, estimate your expected token volume at peak and confirm the provider’s limits can support it. For commercial APIs, scaling beyond those limits is a capacity and cost negotiation with the provider. For open-source models, the constraint shifts entirely. You are no longer managing API limits but by GPU infrastructure. The ceiling is determined by the hardware you own and operate, and expanding it is an infrastructure problem.
LLM Data Security: Compliance, Residency, and Data Retention
Data security decisions narrow down your available options. If your data cannot leave your infrastructure, you are looking at open-source models deployed on-premise.
If cloud deployment is acceptable, three considerations need explicit answers before you commit to a provider.
Data residency is the first. Where is your data physically processed and stored? Some regulatory environments require it to remain within specific geographic boundaries, and your provider needs to be able to demonstrate that.
Regulatory compliance is the second. Industry-specific requirements like HIPAA in healthcare or GDPR in Europe impose concrete obligations on how data is handled. Verify that your provider holds the relevant certifications and can sign the necessary agreements.
The third is zero data retention (ZDR). Most major commercial providers offer this at the API tier. It means the provider does not store or log your prompts and completions, and does not use them to train future models. Confirm ZDR is available for the specific model and pricing tier you are evaluating. It is not always the default.
Multimodal Support and Fine-Tuning Constraints
Modality refers to the types of input your application accepts and the types of output it produces. Text is the baseline. Beyond that, your use case may require users to upload images, submit audio, or receive spoken responses. Not every model supports every modality, and support varies significantly in depth and quality across providers. Start by matching your product’s interaction requirements to what the model can actually handle at inference time.
Modality is also a concern if your use case requires customizing and fine-tuning model behavior beyond what prompting can achieve. Most fine-tuning capability exists at the text layer. A model that processes images or audio at inference time may offer no fine-tuning support for those modalities at all. Verify what is actually supported before assuming it is possible. If multimodal fine-tuning is a requirement and your provider does not support it, the practical path is often a capable base model combined with purpose-built third-party services for the modalities you need: speech-to-text, image recognition, and text-to-speech. Attempting to fine-tune a single model end-to-end is rarely the right starting point.
LLM Cost: Token Pricing, Infrastructure, and Scaling Economics
Cost is evaluated last because it requires context from every factor that comes before it. A cost comparison between two models means nothing if one of them fails your quality threshold, cannot meet your latency requirements, or does not satisfy your security constraints.
The visible cost is token pricing: what you pay per thousand input and output tokens. Output tokens typically cost more than input tokens, and that ratio varies across providers and models. But token pricing is only part of the picture.
Infrastructure cost sits underneath it. For commercial APIs, this includes any additional cloud services required to deploy and operate your solution. For open-source models, it includes GPU hardware, ongoing maintenance, and the engineering cost of keeping the deployment current.
Scaling cost is the third layer. Token costs that look manageable at low volume can become significant as usage grows. Model your costs at different usage thresholds before you commit. What the economics look like at launch, at moderate adoption, and at scale can lead to very different provider decisions. This is also where the commercial versus open-source decision often gets revisited. Commercial APIs charge per token at every level of usage. Open-source models carry fixed infrastructure costs that do not scale with token volume, which means at high enough usage, the economics can shift substantially in favor of self-hosting.
Higher performing models cost more. Stronger security constraints, particularly on-premise deployment, carry significant infrastructure overhead.
How to Apply Your LLM Selection Framework?
Work through the factors in order. Eliminate options at each step. The sequence is a starting point, not a rule. Security moves above quality when compliance is a hard gate, and latency or context window may surface first when they are the dominant constraint for your specific use case. What remains after working through all seven is a defensible shortlist, not a guess.
The framework does not expire at launch. Your product’s requirements will change, old models will get deprecated, new models will raise the quality and cost ceiling, and constraints that did not exist at the start will surface as you scale. Run the same evaluation whenever those conditions shift, whether that means confirming your current model still holds up or making the case for a change.
