{"id":91,"date":"2026-09-16T23:55:53","date_gmt":"2026-09-16T15:55:53","guid":{"rendered":"https:\/\/wp.qoraapi.com\/build-ai-chatbot-api\/"},"modified":"2026-09-20T03:53:31","modified_gmt":"2026-09-19T19:53:31","slug":"build-ai-chatbot-api","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/build-ai-chatbot-api\/","title":{"rendered":"How to Build an AI Chatbot with the API"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">To <strong>build an AI chatbot<\/strong> with the API, you send an ordered list of chat messages to a chat-completions endpoint, stream the reply back to the browser token by token, and resend the conversation history on every turn so the model has context. That is the entire core loop. Memory, retrieval, and cost controls are layers you add on top of it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This tutorial builds the whole thing in order: a first working call, streaming, conversation memory, retrieval-augmented answers, and the production details that decide whether your bot survives its first week of real users.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The core loop, stripped to five steps<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Every chatbot, from a weekend demo to a support agent handling thousands of sessions, runs the same loop. Internalize it and the rest of the build becomes a series of small, obvious additions:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The user types a message in your UI.<\/li>\n<li>Your server appends it to the conversation history as a <code>user<\/code> message.<\/li>\n<li>You POST the full history to <code>\/v1\/chat\/completions<\/code>.<\/li>\n<li>The model returns an assistant message \u2014 streamed token by token, or all at once.<\/li>\n<li>You append that reply to history and render it.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Memory, retrieval, tools, and moderation are all modifications of step 2 or step 3. Nothing in a production chatbot escapes this shape, which is good news: you only have to get one loop right.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1 \u2014 Make one API call work<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Start with the smallest possible script. Use an <strong>OpenAI-compatible endpoint<\/strong> so that every SDK example, framework, and tutorial on the internet works against it unchanged \u2014 only the <code>base_url<\/code> and <code>api_key<\/code> differ. If you have not worked with this interface before, our <a href=\"https:\/\/qoraapi.com\/blog\/openai-compatible-api-guide\/\">OpenAI-compatible API guide<\/a> explains why it became the de facto standard.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install openai\n\n# chatbot.py\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"https:\/\/your-endpoint.example\/v1\",  # one URL, many models\n    api_key=\"YOUR_API_KEY\",\n)\n\ndef reply(history):\n    resp = client.chat.completions.create(\n        model=\"gpt-4o-mini\",          # swap the string to change models\n        messages=history,\n        temperature=0.7,\n    )\n    return resp.choices[0].message.content\n\nhistory = [\n    {\"role\": \"system\", \"content\": \"You are a concise, friendly support assistant.\"},\n    {\"role\": \"user\",   \"content\": \"How do I reset my password?\"},\n]\nprint(reply(history))\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two things are worth noticing here. First, the <code>model<\/code> field is just a string \u2014 you are not locked into a vendor by your code, only by that value. Second, the function is pure: history in, text out. That purity is what makes the later steps easy to add and easy to test.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 2 \u2014 Understand the messages array<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The messages array is the entire state of the conversation. Each entry has a <code>role<\/code> and <code>content<\/code>, and the order matters:<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Role<\/th><th>Who writes it<\/th><th>What it is for<\/th><\/tr><\/thead><tbody><tr><td><code>system<\/code><\/td><td>You<\/td><td>Persona, tone, boundaries, output format. Usually the first message.<\/td><\/tr><tr><td><code>user<\/code><\/td><td>The end user<\/td><td>Questions, instructions, pasted content.<\/td><\/tr><tr><td><code>assistant<\/code><\/td><td>The model<\/td><td>Previous replies \u2014 this is how the bot &#8220;remembers&#8221; what it said.<\/td><\/tr><tr><td><code>tool<\/code><\/td><td>Your code<\/td><td>Results of function calls the model requested.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the part that trips up almost everyone on their first build: <strong>the API is stateless<\/strong>. The model does not remember your last request. If you send only the newest user message, the bot greets you fresh every turn and appears to have amnesia. The illusion of memory exists purely because you resend the whole transcript each time.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That design has one immediate consequence: <em>cost and latency grow with conversation length<\/em>, because you pay for every historical token on every turn. A 40-turn chat re-sends 40 turns of context to answer turn 41. This is the single biggest reason naive chatbots get expensive, and it is why step 4 exists.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 3 \u2014 Stream tokens so the bot feels instant<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A chatbot that pauses for four seconds and then dumps a wall of text feels broken. A chatbot that starts answering in 300 milliseconds feels alive \u2014 even when the total generation time is identical. Streaming is the difference, and it is a small change:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def stream_reply(history):\n    stream = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=history,\n        stream=True,                  # the only new argument\n    )\n    for chunk in stream:\n        delta = chunk.choices[0].delta.content\n        if delta:\n            yield delta               # push each fragment to the UI\n\n# FastAPI: expose the generator as Server-Sent Events\nfrom fastapi import FastAPI\nfrom fastapi.responses import StreamingResponse\n\napp = FastAPI()\n\n@app.post(\"\/chat\")\ndef chat(payload: dict):\n    def events():\n        for piece in stream_reply(payload[\"messages\"]):\n            yield f\"data: {piece}\\n\\n\"   # SSE wire format\n        yield \"data: [DONE]\\n\\n\"\n    return StreamingResponse(events(), media_type=\"text\/event-stream\")\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Streaming has three traps worth knowing before you ship. Reverse proxies and CDNs often buffer responses, which silently destroys the effect. Some frameworks compress the stream, which does the same. And error handling becomes harder, because a failure can arrive <em>after<\/em> you have already rendered half a sentence to the user. Our <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-streaming-sse\/\">streaming and Server-Sent Events guide<\/a> covers the wire format and the proxy-buffering fix in detail.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 4 \u2014 Give the chatbot memory without blowing the budget<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Since you control the history, you control the memory strategy. There are four patterns worth knowing, and most production bots combine two of them:<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Strategy<\/th><th>How it works<\/th><th>Best for<\/th><th>Main cost<\/th><\/tr><\/thead><tbody><tr><td>Full history<\/td><td>Resend every turn<\/td><td>Short sessions, high-stakes accuracy<\/td><td>Cost and latency grow linearly<\/td><\/tr><tr><td>Sliding window<\/td><td>Keep the last N turns<\/td><td>Most chat assistants<\/td><td>Forgets old context abruptly<\/td><\/tr><tr><td>Rolling summary<\/td><td>Summarize older turns into one system message<\/td><td>Long support sessions<\/td><td>Summarization call + detail loss<\/td><\/tr><tr><td>Vector recall<\/td><td>Embed history, retrieve relevant past turns<\/td><td>Long-lived assistants, personalization<\/td><td>Embedding store and extra latency<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A practical default is a sliding window with a token budget rather than a fixed turn count \u2014 trim by measured size, not by guesswork:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def build_context(history, system_prompt, max_tokens=6000):\n    \"\"\"Keep the system prompt and the newest turns that fit the budget.\"\"\"\n    kept, used = [], len(system_prompt) \/\/ 4      # ~4 chars per token\n    for msg in reversed(history):\n        cost = len(msg[\"content\"]) \/\/ 4\n        if used + cost &gt; max_tokens:\n            break\n        kept.append(msg)\n        used += cost\n    return [{\"role\": \"system\", \"content\": system_prompt}] + list(reversed(kept))\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Trim from the middle, never the top: the system prompt defines behavior and the newest turns define the task. Dropping either produces a bot that is suddenly rude or suddenly confused.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 5 \u2014 Add retrieval so the bot can answer about your data<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A pure chat bot only knows what it was trained on plus what you paste in. The moment users ask about your pricing, your internal docs, or last week&#8217;s release notes, it will either refuse or \u2014 worse \u2014 invent an answer. Retrieval-augmented generation fixes this by fetching relevant passages and injecting them into the prompt as context.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The flow is: split your documents into chunks, embed each chunk once, embed the user&#8217;s question at query time, fetch the nearest chunks, and prepend them to the messages array as a system or user message. The chatbot code barely changes \u2014 you are still just assembling a messages array. What changes is where the facts come from.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Getting chunking, embedding models, and re-ranking right is its own discipline; our <a href=\"https:\/\/qoraapi.com\/blog\/ai-embeddings-rag\/\">embeddings and RAG guide<\/a> walks through the pipeline end to end. One rule of thumb from it is worth repeating here: always instruct the model to answer <em>only<\/em> from the retrieved context and to say &#8220;I don&#8217;t know&#8221; otherwise. A chatbot that admits ignorance is far more useful than one that fabricates confidently.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 6 \u2014 Give the bot tools when chat alone is not enough<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once users start asking &#8220;what&#8217;s the status of order 4471?&#8221; the chatbot needs to stop guessing and go look. Tool use (also called function calling) lets the model request a function by name with structured arguments, your code runs it, and you feed the result back as a <code>tool<\/code> message. From the model&#8217;s perspective nothing unusual happened \u2014 the conversation simply gained one more turn.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>tools = [{\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"get_order_status\",\n        \"description\": \"Look up the current status of a customer order.\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\"order_id\": {\"type\": \"string\"}},\n            \"required\": [\"order_id\"],\n        },\n    },\n}]\n\nresp = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=history,\n    tools=tools,\n)\n\ncall = resp.choices[0].message.tool_calls\nif call:\n    order_id = json.loads(call[0].function.arguments)[\"order_id\"]\n    result = get_order_status(order_id)          # your real lookup\n    history.append(resp.choices[0].message)\n    history.append({\n        \"role\": \"tool\",\n        \"tool_call_id\": call[0].id,\n        \"content\": json.dumps(result),\n    })\n    # send again; now the model can answer in natural language\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The pattern to notice is the loop: the model never calls your database, it asks you to. That keeps credentials on your server and makes every action auditable. It also means tool definitions are part of your prompt surface \u2014 vague descriptions produce wrong arguments, and strict JSON schemas produce reliable ones. If you plan to build agents on top of your chatbot, this is the layer that makes them possible.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Choosing a model for a chatbot<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Chat is a forgiving workload. Conversations are short, the quality bar is &#8220;sounds helpful,&#8221; and users tolerate a slightly weaker model far more than they tolerate a two-second pause. That combination makes chat one of the best places to route down a tier:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Small \/ fast models<\/strong> handle greetings, FAQ answers, and simple lookups. Start here and see if users complain \u2014 usually they do not.<\/li>\n<li><strong>Mid-tier models<\/strong> are the right default for open-ended conversation, multi-turn reasoning, and any turn that includes retrieved context.<\/li>\n<li><strong>Frontier models<\/strong> earn their cost only for genuinely hard turns: complex troubleshooting, long document reasoning, or code generation. Route to them per-turn, not per-session.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">A useful trick is to classify the incoming turn cheaply first, then dispatch: a tiny model decides whether this is a greeting, a lookup, or a hard question, and only the last category pays frontier prices. Because the routing decision is made on your side, switching tiers later costs you a config change rather than a rewrite.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Production checklist before you ship<\/h2>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Concern<\/th><th>What to implement<\/th><\/tr><\/thead><tbody><tr><td>Cost control<\/td><td>Token budget per session, cheap model for short turns, caching for repeated questions<\/td><\/tr><tr><td>Rate limits<\/td><td>Retry with exponential backoff on 429, fallback model in the chain<\/td><\/tr><tr><td>Latency<\/td><td>Stream from the first token; never block the UI on a full response<\/td><\/tr><tr><td>Safety<\/td><td>Input moderation, output filtering, a system prompt that defines refusal behavior<\/td><\/tr><tr><td>Observability<\/td><td>Log model, token counts, latency, and error type per request<\/td><\/tr><tr><td>Persistence<\/td><td>Store transcripts server-side; the client should never be the source of truth<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The observability row matters more than it looks. Once you log token counts and latency per request, you can answer questions like &#8220;which users cost the most&#8221; and &#8220;which model actually feels fastest&#8221; from data instead of opinion.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Test the bot before users do<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Chatbots are unusually easy to test badly, because a demo that answers three questions well feels finished. Build a small regression set instead: 20 to 30 real questions with the answer you would accept, stored as plain text. Run them after every prompt change and check two things \u2014 did the bot answer correctly, and did it refuse where it should have refused.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The second check is the one teams skip. A prompt tweak that makes the bot more helpful often makes it more willing to answer questions it has no data for, and that regression will not show up until a customer acts on a fabricated answer. Track refusal behavior alongside accuracy, and re-run the set whenever you change the system prompt, the model string, or the retrieval configuration. Thirty minutes of setup saves you from shipping a bot that confidently invents your refund policy.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Five mistakes that break first chatbots<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Forgetting statelessness.<\/strong> Sending only the newest message and wondering why the bot forgets everything.<\/li>\n<li><strong>Unbounded history.<\/strong> Letting a session grow to 200 turns and paying for all of it on every request.<\/li>\n<li><strong>Blocking on the full response.<\/strong> Skipping streaming and losing the perception of speed you already paid for.<\/li>\n<li><strong>Hard-coding one model.<\/strong> A single model string makes provider changes a refactor instead of a config edit.<\/li>\n<li><strong>No fallback on 429.<\/strong> One rate limit becomes a broken feature in front of the user.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">All five are cheap to fix at build time and expensive to fix in production. Most of them disappear entirely if your chatbot talks to a unified, OpenAI-compatible gateway instead of a single vendor&#8217;s SDK \u2014 one base URL, one key, and model changes become a string edit. <a href=\"https:\/\/qoraapi.com\/\" target=\"_blank\" rel=\"noopener\">qoraapi.com<\/a> is one such relay, exposing many models behind a single endpoint.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Do I need a framework like LangChain to build a chatbot?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. The core loop \u2014 build a messages array, call the API, append the reply \u2014 is roughly 20 lines and is easier to debug without a framework. Reach for a framework when you need multi-step agents, tool orchestration, or built-in tracing, and keep the plain loop for everything simpler.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How do I keep conversation history if the API has no memory?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Store the transcript yourself \u2014 a database row per session, or even a JSON column \u2014 and resend the relevant portion on every request. The API is stateless by design; persistence is your responsibility. Trim with a sliding window or rolling summary once sessions get long.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is the cheapest way to run a chatbot at scale?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Three levers, in order of impact: route simple turns to a small fast model, cap the context you resend per turn, and cache answers to repeated questions. Together they typically cut spend by more than half without any change to the user experience. Our <a href=\"https:\/\/qoraapi.com\/blog\/reduce-ai-api-costs\/\">AI API cost reduction guide<\/a> covers the mechanics.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should the system prompt come first or last?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">First, always. Put the persona, tone, and hard rules in the leading system message, then user and assistant turns after it. Some teams repeat a short version of the key rules at the very end for long contexts, which measurably improves instruction adherence in extended sessions.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Building an AI chatbot with the API is a five-step loop, not a research project. Get one call working against an OpenAI-compatible endpoint, learn that the messages array <em>is<\/em> the memory, add streaming for perceived speed, cap the context so costs stay sane, and layer retrieval on top when the bot needs to know about your data. Each step is independently shippable, so you can put a working bot in front of users long before the last one lands.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ship the loop first, then improve it. Start with the <a href=\"https:\/\/qoraapi.com\/blog\/openai-compatible-api-guide\/\">OpenAI-compatible API guide<\/a> for the request shape, the <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-streaming-sse\/\">streaming guide<\/a> for the UI layer, and the <a href=\"https:\/\/qoraapi.com\/blog\/ai-embeddings-rag\/\">RAG guide<\/a> when your bot needs to answer about your own content.<\/p>\n\n\n\n\n<h3 class=\"wp-block-heading\">Related reading<\/h3>\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-api-streaming-sse\/\">AI API Streaming Explained: How SSE Works and How to Consume It<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-function-calling-tool-use\/\">AI Function Calling Explained: Tools, JSON Schema, and the Tool-Use Loop<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/streaming-chat-ui-react\/\">Building a Streaming Chat UI in React: Patterns for SSE Responses<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/how-to-integrate-ai-api\/\">How to Integrate an AI API into Your Application<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-data-privacy-gdpr\/\">AI API Data Privacy &#038; GDPR: Residency, Logging, and Keeping Prompts Safe<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/local-llm-vs-api\/\">Local LLMs vs API: A Real Cost and Latency Comparison for 2026<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/model-context-protocol-mcp\/\">What Is the Model Context Protocol (MCP)? Connect Your AI to Real Tools<\/a><\/li><\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>A step-by-step tutorial on how to build an AI chatbot with the API: the messages array, streaming tokens, conversation memory, retrieval-augmented answers, and the production details that keep costs sane.<\/p>\n","protected":false},"author":1,"featured_media":89,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,6,9,7],"class_list":["post-91","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-api","tag-ai-api","tag-api-gateway","tag-developer-tools","tag-developers"],"_links":{"self":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/91","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/comments?post=91"}],"version-history":[{"count":2,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/91\/revisions"}],"predecessor-version":[{"id":261,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/91\/revisions\/261"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media\/89"}],"wp:attachment":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media?parent=91"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/categories?post=91"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/tags?post=91"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}