How to Build an AI Chatbot with the API

To build an AI chatbot with the API, you send an ordered list of chat messages to a chat-completions endpoint, stream the reply back to the browser token by token, and resend the conversation history on every turn so the model has context. That is the entire core loop. Memory, retrieval, and cost controls are layers you add on top of it.

This tutorial builds the whole thing in order: a first working call, streaming, conversation memory, retrieval-augmented answers, and the production details that decide whether your bot survives its first week of real users.

The core loop, stripped to five steps

Every chatbot, from a weekend demo to a support agent handling thousands of sessions, runs the same loop. Internalize it and the rest of the build becomes a series of small, obvious additions:

• The user types a message in your UI.
• Your server appends it to the conversation history as a user message.
• You POST the full history to /v1/chat/completions.
• The model returns an assistant message — streamed token by token, or all at once.
• You append that reply to history and render it.

Memory, retrieval, tools, and moderation are all modifications of step 2 or step 3. Nothing in a production chatbot escapes this shape, which is good news: you only have to get one loop right.

Step 1 — Make one API call work

Start with the smallest possible script. Use an OpenAI-compatible endpoint so that every SDK example, framework, and tutorial on the internet works against it unchanged — only the base_url and api_key differ. If you have not worked with this interface before, our OpenAI-compatible API guide explains why it became the de facto standard.

pip install openai

# chatbot.py
from openai import OpenAI

client = OpenAI(
    base_url="https://your-endpoint.example/v1",  # one URL, many models
    api_key="YOUR_API_KEY",
)

def reply(history):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",          # swap the string to change models
        messages=history,
        temperature=0.7,
    )
    return resp.choices[0].message.content

history = [
    {"role": "system", "content": "You are a concise, friendly support assistant."},
    {"role": "user",   "content": "How do I reset my password?"},
]
print(reply(history))

Two things are worth noticing here. First, the model field is just a string — you are not locked into a vendor by your code, only by that value. Second, the function is pure: history in, text out. That purity is what makes the later steps easy to add and easy to test.

Step 2 — Understand the messages array

The messages array is the entire state of the conversation. Each entry has a role and content, and the order matters:

Role	Who writes it	What it is for
system	You	Persona, tone, boundaries, output format. Usually the first message.
user	The end user	Questions, instructions, pasted content.
assistant	The model	Previous replies — this is how the bot “remembers” what it said.
tool	Your code	Results of function calls the model requested.

Here is the part that trips up almost everyone on their first build: the API is stateless. The model does not remember your last request. If you send only the newest user message, the bot greets you fresh every turn and appears to have amnesia. The illusion of memory exists purely because you resend the whole transcript each time.

That design has one immediate consequence: cost and latency grow with conversation length, because you pay for every historical token on every turn. A 40-turn chat re-sends 40 turns of context to answer turn 41. This is the single biggest reason naive chatbots get expensive, and it is why step 4 exists.

Step 3 — Stream tokens so the bot feels instant

A chatbot that pauses for four seconds and then dumps a wall of text feels broken. A chatbot that starts answering in 300 milliseconds feels alive — even when the total generation time is identical. Streaming is the difference, and it is a small change:

def stream_reply(history):
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=history,
        stream=True,                  # the only new argument
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            yield delta               # push each fragment to the UI

# FastAPI: expose the generator as Server-Sent Events
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/chat")
def chat(payload: dict):
    def events():
        for piece in stream_reply(payload["messages"]):
            yield f"data: {piece}\n\n"   # SSE wire format
        yield "data: [DONE]\n\n"
    return StreamingResponse(events(), media_type="text/event-stream")

Streaming has three traps worth knowing before you ship. Reverse proxies and CDNs often buffer responses, which silently destroys the effect. Some frameworks compress the stream, which does the same. And error handling becomes harder, because a failure can arrive after you have already rendered half a sentence to the user. Our streaming and Server-Sent Events guide covers the wire format and the proxy-buffering fix in detail.

Step 4 — Give the chatbot memory without blowing the budget

Since you control the history, you control the memory strategy. There are four patterns worth knowing, and most production bots combine two of them:

Strategy	How it works	Best for	Main cost
Full history	Resend every turn	Short sessions, high-stakes accuracy	Cost and latency grow linearly
Sliding window	Keep the last N turns	Most chat assistants	Forgets old context abruptly
Rolling summary	Summarize older turns into one system message	Long support sessions	Summarization call + detail loss
Vector recall	Embed history, retrieve relevant past turns	Long-lived assistants, personalization	Embedding store and extra latency

A practical default is a sliding window with a token budget rather than a fixed turn count — trim by measured size, not by guesswork:

def build_context(history, system_prompt, max_tokens=6000):
    """Keep the system prompt and the newest turns that fit the budget."""
    kept, used = [], len(system_prompt) // 4      # ~4 chars per token
    for msg in reversed(history):
        cost = len(msg["content"]) // 4
        if used + cost > max_tokens:
            break
        kept.append(msg)
        used += cost
    return [{"role": "system", "content": system_prompt}] + list(reversed(kept))

Trim from the middle, never the top: the system prompt defines behavior and the newest turns define the task. Dropping either produces a bot that is suddenly rude or suddenly confused.

Step 5 — Add retrieval so the bot can answer about your data

A pure chat bot only knows what it was trained on plus what you paste in. The moment users ask about your pricing, your internal docs, or last week’s release notes, it will either refuse or — worse — invent an answer. Retrieval-augmented generation fixes this by fetching relevant passages and injecting them into the prompt as context.

The flow is: split your documents into chunks, embed each chunk once, embed the user’s question at query time, fetch the nearest chunks, and prepend them to the messages array as a system or user message. The chatbot code barely changes — you are still just assembling a messages array. What changes is where the facts come from.

Getting chunking, embedding models, and re-ranking right is its own discipline; our embeddings and RAG guide walks through the pipeline end to end. One rule of thumb from it is worth repeating here: always instruct the model to answer only from the retrieved context and to say “I don’t know” otherwise. A chatbot that admits ignorance is far more useful than one that fabricates confidently.

Step 6 — Give the bot tools when chat alone is not enough

Once users start asking “what’s the status of order 4471?” the chatbot needs to stop guessing and go look. Tool use (also called function calling) lets the model request a function by name with structured arguments, your code runs it, and you feed the result back as a tool message. From the model’s perspective nothing unusual happened — the conversation simply gained one more turn.

tools = [{
    "type": "function",
    "function": {
        "name": "get_order_status",
        "description": "Look up the current status of a customer order.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
    },
}]

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=history,
    tools=tools,
)

call = resp.choices[0].message.tool_calls
if call:
    order_id = json.loads(call[0].function.arguments)["order_id"]
    result = get_order_status(order_id)          # your real lookup
    history.append(resp.choices[0].message)
    history.append({
        "role": "tool",
        "tool_call_id": call[0].id,
        "content": json.dumps(result),
    })
    # send again; now the model can answer in natural language

The pattern to notice is the loop: the model never calls your database, it asks you to. That keeps credentials on your server and makes every action auditable. It also means tool definitions are part of your prompt surface — vague descriptions produce wrong arguments, and strict JSON schemas produce reliable ones. If you plan to build agents on top of your chatbot, this is the layer that makes them possible.

Choosing a model for a chatbot

Chat is a forgiving workload. Conversations are short, the quality bar is “sounds helpful,” and users tolerate a slightly weaker model far more than they tolerate a two-second pause. That combination makes chat one of the best places to route down a tier:

• Small / fast models handle greetings, FAQ answers, and simple lookups. Start here and see if users complain — usually they do not.
• Mid-tier models are the right default for open-ended conversation, multi-turn reasoning, and any turn that includes retrieved context.
• Frontier models earn their cost only for genuinely hard turns: complex troubleshooting, long document reasoning, or code generation. Route to them per-turn, not per-session.

A useful trick is to classify the incoming turn cheaply first, then dispatch: a tiny model decides whether this is a greeting, a lookup, or a hard question, and only the last category pays frontier prices. Because the routing decision is made on your side, switching tiers later costs you a config change rather than a rewrite.

Production checklist before you ship

Concern	What to implement
Cost control	Token budget per session, cheap model for short turns, caching for repeated questions
Rate limits	Retry with exponential backoff on 429, fallback model in the chain
Latency	Stream from the first token; never block the UI on a full response
Safety	Input moderation, output filtering, a system prompt that defines refusal behavior
Observability	Log model, token counts, latency, and error type per request
Persistence	Store transcripts server-side; the client should never be the source of truth

The observability row matters more than it looks. Once you log token counts and latency per request, you can answer questions like “which users cost the most” and “which model actually feels fastest” from data instead of opinion.

Test the bot before users do

Chatbots are unusually easy to test badly, because a demo that answers three questions well feels finished. Build a small regression set instead: 20 to 30 real questions with the answer you would accept, stored as plain text. Run them after every prompt change and check two things — did the bot answer correctly, and did it refuse where it should have refused.

The second check is the one teams skip. A prompt tweak that makes the bot more helpful often makes it more willing to answer questions it has no data for, and that regression will not show up until a customer acts on a fabricated answer. Track refusal behavior alongside accuracy, and re-run the set whenever you change the system prompt, the model string, or the retrieval configuration. Thirty minutes of setup saves you from shipping a bot that confidently invents your refund policy.

Five mistakes that break first chatbots

• Forgetting statelessness. Sending only the newest message and wondering why the bot forgets everything.
• Unbounded history. Letting a session grow to 200 turns and paying for all of it on every request.
• Blocking on the full response. Skipping streaming and losing the perception of speed you already paid for.
• Hard-coding one model. A single model string makes provider changes a refactor instead of a config edit.
• No fallback on 429. One rate limit becomes a broken feature in front of the user.

All five are cheap to fix at build time and expensive to fix in production. Most of them disappear entirely if your chatbot talks to a unified, OpenAI-compatible gateway instead of a single vendor’s SDK — one base URL, one key, and model changes become a string edit. qoraapi.com is one such relay, exposing many models behind a single endpoint.

Frequently asked questions

Do I need a framework like LangChain to build a chatbot?

No. The core loop — build a messages array, call the API, append the reply — is roughly 20 lines and is easier to debug without a framework. Reach for a framework when you need multi-step agents, tool orchestration, or built-in tracing, and keep the plain loop for everything simpler.

How do I keep conversation history if the API has no memory?

Store the transcript yourself — a database row per session, or even a JSON column — and resend the relevant portion on every request. The API is stateless by design; persistence is your responsibility. Trim with a sliding window or rolling summary once sessions get long.

What is the cheapest way to run a chatbot at scale?

Three levers, in order of impact: route simple turns to a small fast model, cap the context you resend per turn, and cache answers to repeated questions. Together they typically cut spend by more than half without any change to the user experience. Our AI API cost reduction guide covers the mechanics.

Should the system prompt come first or last?

First, always. Put the persona, tone, and hard rules in the leading system message, then user and assistant turns after it. Some teams repeat a short version of the key rules at the very end for long contexts, which measurably improves instruction adherence in extended sessions.

Conclusion

Building an AI chatbot with the API is a five-step loop, not a research project. Get one call working against an OpenAI-compatible endpoint, learn that the messages array is the memory, add streaming for perceived speed, cap the context so costs stay sane, and layer retrieval on top when the bot needs to know about your data. Each step is independently shippable, so you can put a working bot in front of users long before the last one lands.

Ship the loop first, then improve it. Start with the OpenAI-compatible API guide for the request shape, the streaming guide for the UI layer, and the RAG guide when your bot needs to answer about your own content.

Related reading

• AI API Streaming Explained: How SSE Works and How to Consume It
• AI Function Calling Explained: Tools, JSON Schema, and the Tool-Use Loop
• Building a Streaming Chat UI in React: Patterns for SSE Responses
• How to Integrate an AI API into Your Application
• AI API Data Privacy & GDPR: Residency, Logging, and Keeping Prompts Safe
• Local LLMs vs API: A Real Cost and Latency Comparison for 2026
• What Is the Model Context Protocol (MCP)? Connect Your AI to Real Tools