Self-host an LLM only when you can keep a GPU above roughly 8% utilization on batchable traffic. Call an API when your traffic is interactive, spiky, or needs frontier reasoning. Below about 300,000 requests per month, the engineering time you spend running the box costs more than the tokens you save.
The decision isn’t “better” — it’s “for which workload”
Every product has two workloads wearing one name. Interactive: a human is waiting, concurrency is bursty, and requests arrive at 9:14 a.m. because that is when users open the app. Batch: classification, embedding, redaction, map-step summarization, eval judging — work that can sit in a queue for ten seconds without anyone noticing.
- API = variable cost, zero fixed cost. You pay per token, capacity is someone else’s problem, and an idle month costs nothing.
- Local = fixed cost, near-zero marginal cost. The card bills you whether it serves ten requests or ten million. Capacity is now your problem.
The classic failure: benchmark quality on a public leaderboard, buy a GPU, then discover the real traffic profile is 40 concurrent chat sessions at 9 a.m. and nothing at 3 a.m. A flat bill against a spiky load is a bad trade at any price per token.
So replace “which is better” with three questions:
- Can this work be queued and batched? If yes, local has a real shot. If no, you are buying an expensive queue.
- Does your own eval show a quality delta on your task? Not a public leaderboard — your golden set, your grader.
- Do you have a non-cost reason to own the weights? Data residency, an air-gapped deployment, a contractual ban on third-party processing, or a fine-tune you cannot ship anywhere else.
Three noes means call the API. Any yes moves you into the hybrid design below.
Cost model: per-token API vs per-GPU-hour local
Model prices move every few months, so build the model in ratios and it stays valid. Two variables are enough: P, the blended API price per 1M tokens (input and output mixed at your real ratio), and G, the rental cost of one GPU-hour on a card that comfortably serves a small open-weight model.
Assume a typical request of 2,500 blended tokens. Then:
API cost per request = 2,500 / 1,000,000 * P = 0.0025 P
Local cost per month = G * 730 hours = 730 G (per card, flat)
Break-even N (requests/month) = 730 G / 0.0025 P = 292,000 * (G / P)When one GPU-hour costs roughly the same as 1M blended API tokens, break-even lands at about 292,000 requests per month per card — call it 300k. That number alone is not the answer, because it ignores capacity. One card has a ceiling:
- Interactive concurrency (1–4 sequences in flight, no batching benefit): roughly 7 requests/minute sustained, about 300k requests/month.
- Continuous batching (32–64 sequences in flight): aggregate throughput rises 8–12×, roughly 84 requests/minute, about 3.6M requests/month.
Put the two together and the break-even volume sits exactly at interactive capacity. That is the whole story of local LLM economics:
| Monthly requests (≈2.5k tokens each) | API cost index (P=1) | Cards @ interactive (≈7 req/min) | Local cost index (730/card) | Cards @ batched (≈84 req/min) | Local cost index | Cheaper lane |
|---|---|---|---|---|---|---|
| 50,000 | 125 | 1 (minimum) | 730 | 1 (minimum) | 730 | API |
| 150,000 | 375 | 1 | 730 | 1 | 730 | API |
| 300,000 | 750 | 1 | 730 | 1 | 730 | Wash |
| 750,000 | 1,875 | 3 | 2,190 | 1 | 730 | Local (batched only) |
| 2,000,000 | 5,000 | 7 | 5,110 | 1 | 730 | Local |
| 5,000,000 | 12,500 | 17 | 12,410 | 2 | 1,460 | Local |
Read it as a utilization problem, not a price problem. At interactive concurrency, local only pulls ahead once you are already saturated — and then you are one card away from an outage. At batch concurrency, local wins from about 8% utilization upward.
Two costs the index column omits. Engineering time: 0.2–0.5 FTE for one model, one region, one runtime, which in most markets equals 2–6 card-months per year — so below roughly 1M requests/month it usually exceeds the GPU bill. Redundancy: your availability ceiling is one card’s availability, and a second availability zone doubles the fixed cost. For the API-side levers — caching, token budgeting, context trimming — see our guide to reduce AI API costs.
Latency reality: TTFT, cold starts, and the batching trap
Every request is queueing + prefill + decode, and each behaves differently per lane.
Prefill is compute-bound and scales with prompt length, which produces the most counterintuitive result here: a hosted mid-tier API usually beats a single-card local 8B model on time-to-first-token for long prompts. A 4-bit 8B model on one card prefills at roughly 1,000–2,000 tokens/second, so a 4,000-token RAG prompt costs 2–4 seconds before the first token appears, while a hosted provider running that same prefill on a large, well-optimized fleet lands at 0.3–0.8 seconds. If your product is chat with retrieved context, the local lane feels slower at the start of every answer.
Decode is where local wins. A 4-bit small model decodes faster per stream than a frontier model, often by 1.5–3×, so the crossover is a function of output length. Short outputs such as classification or extraction favor the API lane, because prefill dominates and the API prefills better. Long outputs of 600+ tokens with short prompts favor local, because decode dominates.
Cold starts are minutes, not seconds. An 8B model at 4-bit is roughly 5 GB of weights: 3–8 seconds from local NVMe, 15–40 seconds from network storage, plus 5–20 seconds of CUDA graph capture and warmup before throughput stabilizes. Add a serverless GPU provider and you also pay image pull and model download — realistically 1–5 minutes. So any interactive local service needs a warm replica at all times: scale-to-zero is off the table and you pay 24/7 whether or not traffic shows up.
The batching trap. Continuous batching raises aggregate throughput 8–12× but adds queueing delay per request. With 64 sequences in flight, TTFT can grow by 1–3 seconds even as throughput improves dramatically. High throughput and low latency are different operating points on the same hardware, so never serve interactive and batch traffic from one pool.
Two effects worth measuring, not assuming. Prefix caching cuts TTFT and cost when requests share a fixed preamble plus retrieved context, and both lanes support it. KV cache pressure couples latency to quality: at 32k+ contexts you either lower max_model_len and silently truncate, or quantize the KV cache and lose accuracy.
Quality & capability gap: when it actually matters
“Open-weight models are 90% as good” is a meaningless sentence, because the gap is not uniform across tasks. Sort your workload into three buckets and the decision mostly makes itself:
| Task bucket | Examples | Observed gap vs frontier | Verdict |
|---|---|---|---|
| Well-specified, single-step | Classification, schema extraction, PII redaction, short summaries, embeddings, reranking | Near zero on a task-specific eval | Local lane |
| Multi-step but bounded | RAG answer synthesis with citations, rewriting, translation | Moderate; needs constrained decoding and retries | Local with API fallback |
| Compounding reasoning | Agentic tool loops, code that must compile, math, cross-document synthesis | Large, and multiplies with step count | API lane only |
The mechanism behind the third row is error compounding: a model that is 95% reliable per step is roughly 60% reliable over ten steps. Frontier models earn their price in exactly this regime, and no amount of prompt engineering closes it on a small quantized model.
Quantization has a task-dependent cost. Moving from FP16 to 4-bit weights costs measurable accuracy on reasoning-heavy work and almost nothing on classification and extraction. Use your own eval as the gate: if the delta is under about 2 points on your task set, quantize and take the memory savings; if it is over about 5 points, do not ship that model locally for that task.
The real reason to self-host is distillation, not deployment. Run the hard task on a frontier model, filter the outputs with a verifier, then fine-tune a small open-weight model on the accepted pairs and serve that locally. This converts a permanent quality gap into a one-time training cost, and it is the only strategy where local wins on quality and cost at once — for a narrow, high-volume task. It also creates a genuine reason to own the weights, which a plain deployment never does.
One hard rule follows: if the task is not in your eval harness, you cannot self-host it. Build a 200-example golden set with a programmatic grader before you provision a GPU, or you will have no way to detect the day a quantization change quietly degrades production.
Non-cost reasons are decisive too: data residency, air-gapped deployments, contracts that forbid third-party processing. With one of those, the cost comparison is moot.
Ops burden: what you actually sign up for
Pick the runtime by workload shape, not by GitHub stars:
- llama.cpp (GGUF) — fastest path to “it runs” on CPU or Apple silicon, heavily optimized for single-stream inference. No paged attention, so multi-tenant throughput is poor. Right for desktop and edge, wrong for a multi-tenant service.
- Ollama — llama.cpp plus a model registry and an OpenAI-compatible endpoint. Excellent for local development and prototypes; limited control over batching and memory, which is why it is usually not the production serving layer.
- vLLM — paged attention, continuous batching, and an OpenAI-compatible server. The default for production. You will tune
gpu_memory_utilization,max_num_seqs,max_model_len, and tensor-parallel size, and you will learn what a KV-cache OOM looks like. SGLang is worth evaluating if your traffic is prefix-heavy. - TensorRT-LLM — highest throughput ceiling, highest build complexity, least portability. Justified only if inference is your product.
GPU provisioning is a capacity problem, not a pricing problem. Mid-range accelerators go out of stock regionally for hours, which can force you into a different region than your application and add cross-region latency to every request — a cost that never appears on the invoice. Reserved capacity lowers the hourly rate but converts variable cost into a fixed monthly commitment, removing the advantage you were chasing.
Autoscaling does not work the way web autoscaling does. With cold starts measured in minutes, scale-to-zero is unusable for interactive traffic. Scale on queue depth, not CPU or GPU utilization — utilization is a lagging signal that reacts after users are already waiting. The pattern that survives production is one always-warm replica plus overflow to the API lane, which is also the cheapest way to absorb the 9 a.m. spike you sized for above.
Upgrades are a standing tax. Every new model release means re-quantizing weights, re-validating quality, re-tuning max_model_len and batch sizes, and re-running the eval suite. CUDA, driver, and vLLM version drift adds a recurring compatibility chore, and you now own the 3 a.m. page for OOM and CUDA errors that a hosted provider absorbs for you.
Hybrid: route by task, overflow by load
The hybrid design dominates both pure options in most products, because the two lanes are good at different things. Assign tasks deliberately rather than by preference:
| Task class | Lane | Why |
|---|---|---|
| PII redaction, classification, tagging | Local | High volume, low stakes, short outputs — prefill-bound work that batches well |
| Embeddings for RAG | Local | Token volume dwarfs everything else and the quality gap is negligible |
| Map-step summarization of long documents | Local | Queueable, latency-tolerant, enormous token count |
| Eval judging at scale | Local | Cheap enough to grade every production request |
| Draft generation for review | Local | A human or a stronger model reviews before it ships |
| Final user-facing answer | API | Wrong answers cost money and trust |
| Agentic tool loops, code, math | API | Errors compound per step |
| Long-context synthesis (32k+) | API | KV-cache pressure hurts local latency and quality at once |
| Anything not in your eval set | API | You cannot self-host what you cannot measure |
Fallback direction matters as much as lane assignment. The batch lane is local-first: when the card saturates, spill to the API instead of queueing, because the work is not urgent. The interactive lane is API-first: a local cold start cannot absorb a spike, so use local only for the short-prompt, long-output shape it wins on. Getting this backwards turns a latency win into timeouts.
Here is a router small enough to read in one sitting. It decides by task class, then by prompt length and queue depth, and it never lets the local lane hard-fail a feature:
# Route by task class, then overflow by load. No vendor loyalty.
import os
from openai import OpenAI
# Both lanes speak the same OpenAI-compatible wire format.
LOCAL = OpenAI(base_url=os.environ["LOCAL_BASE_URL"], api_key="local") # vLLM / Ollama
API = OpenAI(base_url="https://qoraapi.com/v1", api_key=os.environ["QORA_KEY"])
LOCAL_TASKS = {"classify", "embed", "redact", "tag", "draft", "map_summarize"}
API_TASKS = {"reason", "code", "agent", "synthesize", "final_answer"}
LOCAL_MAX_PROMPT_TOKENS = 6000 # long prompts blow up local prefill TTFT
LOCAL_MAX_QUEUE = 8 # saturated card -> spill, do not queue a user
def pick_lane(task: str, prompt_tokens: int, queue_depth: int) -> str:
if task in API_TASKS:
return "api"
if prompt_tokens > LOCAL_MAX_PROMPT_TOKENS or queue_depth > LOCAL_MAX_QUEUE:
return "api" # overflow, not failure
return "local"
def complete(task, messages, prompt_tokens, queue_depth, **kwargs):
lane = pick_lane(task, prompt_tokens, queue_depth)
client, model = (LOCAL, "local-8b-instruct") if lane == "local" \
else (API, "mid-tier-model")
try:
return client.chat.completions.create(
model=model, messages=messages, **kwargs)
except Exception:
if lane == "local": # local never breaks a feature
return API.chat.completions.create(
model="mid-tier-model", messages=messages, **kwargs)
raiseLog the chosen lane on every request. That log is your utilization metric, your cost attribution, and the evidence you need to move a task between lanes — the discipline described in our guide to model routing. Once the lanes are interchangeable at the code level, moving a task is a one-line change instead of a migration.
A gateway that mixes both behind one key
The router above is only cheap if the lanes are interchangeable at the code level, and that is the practical argument for the OpenAI-compatible API wire format. Your local vLLM or Ollama server already exposes /v1/chat/completions, so the local lane needs no bespoke client. On the hosted side, a relay gives you the same shape across many providers: one endpoint, one credential, and a model string instead of five SDKs and five key rotations.
That is what qoraapi.com provides — an AI API relay that fronts many hosted models behind one OpenAI-compatible key. In the router above, the only difference between the two branches is a base_url and a model name, so your local baseline and your hosted fallback run through the same code path, the same retry logic, and the same eval harness. A model deprecation becomes a config change, a provider outage becomes a fallback entry, and A/B-testing a hosted model against your local baseline is a one-line diff. Log the model, lane, token counts, and latency on every call — a hybrid system without per-request attribution becomes an expensive mystery within a quarter.
Frequently asked questions
At what monthly volume does self-hosting actually pay off?
For batchable traffic, break-even is around 300,000 requests per month per card — only about 8% of a card’s batched capacity, so local wins comfortably above that. For interactive traffic, break-even roughly equals the card’s capacity, meaning you are effectively saturated before you break even. Add 0.2–0.5 FTE of engineering time and the practical threshold for interactive self-hosting is well above 1M requests per month.
Why is my local model slower to first token than a hosted API?
Because TTFT is dominated by prefill, and prefill is compute-bound. One card prefills a 4,000-token prompt at roughly 1,000–2,000 tokens/second, so you wait 2–4 seconds for the first token, while a hosted provider runs that same prefill on a much larger, better-optimized fleet and returns in under a second. Local wins on decode speed, not prefill — so it feels fast for long outputs and slow for short ones.
Can I run a large open-weight model locally instead?
Yes, but the math turns unfavorable. Multi-GPU tensor parallelism roughly doubles fixed cost without doubling throughput, because inter-GPU communication becomes the bottleneck, and quantizing a large model to fit still leaves it behind frontier models on agentic and multi-step reasoning. Large local models make sense for data residency; for cost, a small model on a batched lane usually wins.
Do I need to fine-tune to justify self-hosting?
Not strictly, but it is the strongest justification. Serving a stock open-weight model locally is a pure cost-versus-ops comparison that is hard to win at moderate volume. Distilling frontier outputs into a small local model for one narrow task converts a permanent quality gap into a one-time training cost — and that is where local reliably beats the API on both axes.
Conclusion
The build-or-buy question resolves into arithmetic and workload shape. Compute your break-even as 292,000 × (G / P) requests per month per card, then compare it against real capacity at the concurrency you actually run. If the work batches, local wins early and by a wide margin. If it is interactive and spiky, the API lane wins on total cost of ownership long before it wins on price per token — because the GPU is the smallest part of what self-hosting costs.
Then stop treating it as a binary. Put batchable, low-stakes, high-volume work on the local lane, keep hard reasoning and user-facing answers on the API lane, and expose both through one OpenAI-compatible interface so switching lanes is a config change.
Leave a Reply