Adding AI to an existing SaaS is a weekend project, not a quarter-long ML initiative. You need exactly three things: one high-leverage, low-risk feature (summary, semantic search, draft reply, or support triage), a backend route that calls an OpenAI-compatible endpoint, and a gateway so a single key covers every model. No training, no GPUs, no ML hire.
The hard part is not the API call — that is twenty lines of code. The hard part is choosing a feature whose failure mode your users will tolerate, then wrapping it in enough caching, metering, and guardrails that one bad traffic week does not become one bad billing month. This is the order we would actually do it in.
Start with the feature, not the model
Most teams pick a model first and then look for something to point it at. That is backwards, and it is why so many “we added AI” launches stall. Start with a feature that clears four filters:
- The input already lives in your database. If you need a new ingestion pipeline before the AI can run, you have a data project, not a weekend project.
- A human sees the output before it has consequences. Summaries, drafts, and ranked search results get reviewed. Auto-sent emails and autonomous actions do not.
- A wrong answer degrades to “less useful,” not “harmful.” A mediocre summary costs a user five seconds. A wrong refund figure costs you money and trust.
- You can disable it with a flag. If turning the feature off requires a deploy, it is not ready to ship.
Features that clear all four filters almost always fall into one of four shapes. Score them on the effort-versus-value grid before you write a line of prompt code:
| Quadrant | Effort | Value | What belongs here | Action |
|---|---|---|---|---|
| Quick win | Low (1–3 days) | High | Summarize a long record; draft a templated reply; semantic search over docs you already store; support ticket triage | Ship this weekend |
| Filler | Low (1 day) | Low | Auto-tagging, sentiment badges, title suggestions | Do it while eval runs are queued |
| Bet | High (weeks) | High | Agent that takes actions; retrieval over messy multi-source data; per-customer personalization | Prototype behind a flag, plan properly |
| Trap | High (weeks) | Low | Fine-tuning on a few hundred examples; a general-purpose chatbot that answers everything | Skip |
The Trap quadrant is where most first attempts die. Fine-tuning needs thousands of clean labeled examples and a task that will not change next quarter; a general chatbot needs the entire support knowledge base to be accurate before it is useful at all. Neither is a weekend.
Now narrow the Quick win quadrant down to one feature. The four candidates differ in ways that matter more than the model you pick:
| Feature | What you need first | Failure mode | Why it is a good first ship |
|---|---|---|---|
| Summarization | Long text records you already store (tickets, notes, transcripts) | Misses a detail; slightly generic | Pure read-only. Nothing downstream breaks if it is wrong. |
| Semantic search | An embedding index over existing content | Ranks an irrelevant doc first | Users still see real documents, just in a different order. |
| Draft generation | A small set of real examples of the output you want | Tone is off; needs edits | The human edits before sending — the model never has the last word. |
| Support triage | A ticket queue and a category list | Misroutes to the wrong team | Internal-only. Worst case, a human reassigns it in two clicks. |
The architecture in one diagram
Every weekend AI feature has the same shape. Draw it once and the implementation stops being ambiguous:
Browser / mobile client
│ your session cookie only — no provider key ever ships to the client
▼
Your backend
POST /api/summarize ← the feature endpoint you own
│
├─ 1. cache lookup hash(model + prompt version + normalized input)
├─ 2. quota check per-user daily tokens, global circuit breaker
├─ 3. prompt assembly system rules + delimited untrusted user text
├─ 4. one chat() call timeout, one retry, usage logged
└─ 5. degrade path return null, hide the UI, app keeps working
│
▼
AI API relay (one base URL · one key · OpenAI-compatible wire format)
├──► fast/cheap model default for this feature
├──► mid model escalation when the input is long or nuanced
└──► frontier model fallback when the default is throttled
Four invariants make this architecture worth drawing:
- The provider key lives only in server environment variables. A key in frontend code is a key you have already leaked.
- Your backend owns the prompt. If the client can send a system message, a user can rewrite your product’s behavior.
- Every model call goes through one function. Caching, metering, retries, and logging live in that function — not scattered across twelve endpoints.
- The relay is one base URL. Changing which model answers is a config value, not a code change.
Wire an AI API in an afternoon
The integration work is four steps: get a base URL and key, smoke-test with one curl, write one server-side function, expose one route. Smoke-test first — it separates “my code is wrong” from “my credentials are wrong” in about thirty seconds.
curl https://YOUR_GATEWAY_BASE/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AI_API_KEY" \
-d '{
"model": "YOUR_MODEL_ID",
"messages": [{"role": "user", "content": "Reply with the single word: ok"}],
"max_tokens": 5
}'
# Expect: {"choices":[{"message":{"content":"ok",...}}],"usage":{...}}
# If you get 401, the key is wrong. If you get 404, the base URL is missing /v1.
Then put the same call behind your own route. This is the whole feature — the rest is prompt tuning:
// POST /api/summarize — server-side only.
import express from "express";
const app = express();
app.use(express.json({ limit: "1mb" }));
const AI_BASE = process.env.AI_BASE_URL; // one gateway base URL
const AI_KEY = process.env.AI_API_KEY; // server env var, never sent to clients
async function chat(messages, { model, maxTokens = 400, temperature = 0.2 } = {}) {
const res = await fetch(`${AI_BASE}/chat/completions`, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${AI_KEY}`,
},
body: JSON.stringify({ model, messages, max_tokens: maxTokens, temperature }),
signal: AbortSignal.timeout(30_000), // hard ceiling: never hang a request thread
});
if (!res.ok) {
throw new Error(`AI ${res.status}: ${(await res.text()).slice(0, 200)}`);
}
const data = await res.json();
return { text: data.choices[0].message.content, usage: data.usage };
}
app.post("/api/summarize", async (req, res) => {
const doc = String(req.body.text || "").slice(0, 12_000); // cap input, cap cost
if (doc.length < 200) return res.json({ summary: null }); // too short to be worth a call
try {
const { text, usage } = await chat(
[
{ role: "system",
content: "Summarize the input in 3 bullets. Use only facts present in the input. If a fact is uncertain, omit it." },
{ role: "user", content: `<document>\n${doc}\n</document>` },
],
{ model: process.env.AI_MODEL_SUMMARY } // model id is config, not code
);
logUsage(req.user.id, "summarize", usage);
res.json({ summary: text });
} catch (err) {
res.json({ summary: null, degraded: true }); // feature hides itself; the app still works
}
});
Five lines in that snippet are doing more work than they look like they are:
AbortSignal.timeout(30_000)— without it, a slow provider becomes a pile of stuck requests and a memory graph that climbs forever..slice(0, 12_000)— input length is the single biggest cost variable, and it is attacker-controlled. Cap it at the edge of your own route.if (doc.length < 200)— skip the call entirely when the answer is obvious. Cheap features are built from the calls you do not make.process.env.AI_MODEL_SUMMARY— the model is configuration. That is what makes the eval-and-swap loop later free.- The
catchreturningdegraded: true— the feature fails quietly instead of taking your page down with it.
The wire format is the same in Python, Go, PHP, or Ruby, because it is just an HTTP POST with a JSON body. If you want the request and response shape explained field by field, our guide on how to integrate an AI API walks through it.
Match the feature to a use case
All four weekend features hit the same endpoint. What changes is the system prompt, how you assemble the input, how you parse the output, and which model tier you route to. Getting that mapping right is most of the quality difference:
- Summarization — one long document in, short prose out. Cheap/fast tier is usually enough; escalate only when the source is long or legally sensitive. Ask for a fixed shape (three bullets, or a fixed set of fields) so the UI can render it reliably.
- Semantic search — embed the query and the corpus, rank by similarity, then optionally pass the top few chunks to a model to re-rank or answer. Two calls, not one, and the embedding call is the cheap half.
- Draft generation — retrieve two or three real examples of good output from your own history and include them in the prompt. Few-shot beats adjectives: “write in a friendly tone” does less than one real example.
- Support triage — constrain the output to your existing category list and nothing else. A model choosing from eight known labels is far more reliable than one inventing a label.
If you are still deciding which feature is worth building, our breakdown of AI API use cases maps common product surfaces to the technique each one needs. Pick one, ship it, and let real usage tell you which is second.
Keep it cheap and safe
Weekend features turn into production incidents in three predictable ways: the bill, the abuse, and the output. All three are solvable with code you write in the same afternoon.
Cache the output, not the request
Cache the result keyed by a hash of model + prompt version + normalized input. Summaries are unusually cache-friendly because the same record gets reopened many times and only changes occasionally. Invalidate on document edit, and never cache anything personalized to a user — that turns a cache into a data leak.
Meter every call, then cap it
Log tokens, model, feature, and user id on every call. Without that log you cannot answer “which feature costs the most” — and that is the only question that matters when the invoice grows. Then add two limits: a per-user daily token cap, and a global circuit breaker that stops non-critical AI calls when daily spend crosses a threshold.
// Wrap every model call: cache → quota → call → meter.
const key = `sum:${model}:${PROMPT_VERSION}:${sha256(normalize(doc))}`;
const cached = await redis.get(key);
if (cached) { metrics.inc("cache_hit"); return cached; }
if (!(await withinQuota(userId))) throw new QuotaExceeded(); // per-user daily cap
const { text, usage } = await chat(messages, { model });
await redis.set(key, text, "EX", 60 * 60 * 24); // TTL; invalidate on document edit
meter(userId, "summarize", usage); // tokens + model + feature
return text;
Two details make this work. Including PROMPT_VERSION in the key means editing your prompt invalidates the cache automatically instead of silently serving stale output. And max_tokens on every call is your runaway-generation brake — an unbounded completion is the most common single-call cost spike.
Treat both directions as untrusted
Model output is untrusted input. Render it as text, never as HTML; never interpolate it into SQL, a shell command, or a template that can execute. Model input is also untrusted: wrap user text in explicit delimiters, tell the system prompt that content inside those delimiters is data rather than instructions, and keep the system prompt server-side so a client cannot rewrite it. The failure mode to design against is indirect prompt injection — a malicious string hiding in a document your app summarizes. Our guide to AI API security covers the layered defenses; for the cost levers in depth, see how to reduce AI API costs.
The ship checklist
Do not ship without these seven. Each one takes under an hour and each one prevents a specific class of launch-day regret:
| Check | Pass condition | What it prevents |
|---|---|---|
| Eval on a frozen sample | 30–50 real inputs, a written rubric, a recorded score before you touch the prompt again | “It feels better” prompt changes that quietly regress quality |
| Fallback model | A forced 429 returns a degraded UI, not a 500 | A provider outage becoming your outage |
| Prompt versioning | Prompts live in a file with a version string, included in cache keys and logs | Untraceable quality changes and stale cache hits |
| Cost guard | Per-user cap and global breaker, both tested by deliberately tripping them | A single abusive account or a retry loop draining your budget |
| Latency budget | p95 inside your UX threshold, measured, not guessed | A “fast” feature users abandon because it stalls |
| Kill switch | An env flag disables the feature with no deploy | Being unable to stop the bleeding at 2 a.m. |
| Per-call logging | Model, latency, tokens, cache hit/miss, prompt version on every request | Flying blind when cost or quality moves |
One more habit compounds: give users a thumbs-down button, and store the input alongside the negative rating. Within a week you have a real eval set built from actual failures instead of invented test cases — which is far more valuable than any prompt trick you will read about.
Why a gateway beats raw provider keys
Everything above assumes you can change the model without touching application code. Raw provider keys break that assumption. Four provider SDKs mean four auth shapes, four error taxonomies, four retry conventions, and four places to rotate a leaked key. “Switching models” becomes a refactor, so you stop switching — and you stay on the wrong model long after you know it is wrong.
An AI API relay collapses that into one base URL and one key in OpenAI-compatible format. The practical consequences are concrete:
- Model swaps are strings. The same
chat()function above serves a fast model today and a frontier model tomorrow; onlyAI_MODEL_SUMMARYchanges. - Fallbacks become trivial. When every model is reachable through one endpoint, your retry loop does not need provider-specific branches.
- One bill, one meter, one place to cap spend. Cost attribution by feature is a query, not an integration project.
- Smaller blast radius. One credential to rotate instead of four, and it never leaves your server.
This is the difference between a weekend feature and a weekend feature you can still improve in month three. qoraapi.com is an AI API relay that exposes many models behind one OpenAI-compatible key, which is exactly the shape this architecture wants. If you are comparing options, our guide to choosing the best AI API gateway covers the criteria that actually matter — and for the operational side of that switch, the guide to handling 429s and rate limits pairs with it.
Frequently asked questions
Can I ship an AI feature in a weekend without ML experience?
Yes. You are doing integration, not machine learning. There is no training loop, no dataset to label, and no GPU. The skills that matter are the ones you already have: designing an API route, handling errors, caching, and writing a clear system prompt. The ML-specific work — fine-tuning, embedding pipelines at scale, model evaluation research — only becomes relevant after the feature is live and earning its place.
Do I need to fine-tune a model for my domain?
Almost never as a first step. A well-written system prompt with two or three real examples from your own product usually gets you most of the way, and it can be changed in seconds. Fine-tuning is worth revisiting only when you have thousands of clean labeled examples, a task that is stable and high-volume, and evidence that prompt engineering has plateaued. Until all three are true, it is effort spent in the low-value quadrant.
How do I stop AI costs from spiking unexpectedly?
Four controls, in order of impact: cap max_tokens on every call, cap input length at your own route, cache outputs keyed by model plus prompt version plus input, and enforce a per-user daily token limit with a global circuit breaker. Add per-call logging so you can attribute spend to a specific feature — a spike you cannot attribute is a spike you cannot fix.
What happens if the model provider goes down?
Design for degradation, not for uptime guarantees. Retry once against a second model, and if that fails too, return a null result with a degraded flag so the UI simply hides the AI panel and the underlying product keeps working. A summarization feature that disappears for an hour is an inconvenience; a summarization feature that returns 500s takes the page down with it.
Should I stream the response?
Only if the user is waiting on a long generation and the perceived latency matters more than the complexity. Short outputs — three bullets, a category label, a JSON object — are faster to deliver as one response than to stream. If you do need it, the wire format and the proxy-buffering trap are covered in our guide to streaming responses with SSE.
Ship the feature, keep the architecture
Pick one feature from the Quick win quadrant. Put it behind a single server-side route that calls one OpenAI-compatible endpoint through one chat() function. Cap the input, cap the output, cache the result, meter every call, and make the model a config value. Run the seven-item checklist, ship it behind a flag, and let a week of real traffic build your eval set.
That gets you a working AI feature in a weekend — and, more importantly, an architecture where the second feature takes an afternoon instead of another weekend. Because the model was never the hard part.
Leave a Reply