{"id":96,"date":"2026-09-16T23:56:30","date_gmt":"2026-09-16T15:56:30","guid":{"rendered":"https:\/\/wp.qoraapi.com\/multimodal-ai-api\/"},"modified":"2026-09-20T03:53:46","modified_gmt":"2026-09-19T19:53:46","slug":"multimodal-ai-api","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/multimodal-ai-api\/","title":{"rendered":"Multimodal AI APIs: Working with Vision and Audio"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A <strong>multimodal AI API<\/strong> accepts more than text: you send images, audio, and document pages through the same chat-style request you already use, and the model returns text, structured JSON, or tool calls. This guide covers the exact request shapes, the real cost drivers, and the production patterns that survive contact with users.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you have shipped a text-only integration, you already know 90% of what you need. Multimodal did not introduce a new protocol \u2014 it made the <code>content<\/code> field of a message polymorphic. That one change unlocks screenshots, scanned invoices, voice memos, and call recordings through the endpoint you already call.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What &#8220;multimodal&#8221; actually means at the API level<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Marketing copy uses &#8220;multimodal&#8221; loosely, so it helps to be precise about the four capabilities developers actually ship:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Image in, text out (vision).<\/strong> You attach one or more images and ask a question. The model reasons over pixels and layout, not just extracted characters. This is the capability people mean when they say &#8220;vision API&#8221; or refer to GPT-4V-class image understanding.<\/li>\n<li><strong>Audio in, text out (transcription).<\/strong> A dedicated transcription endpoint converts speech to text, usually with timestamps and optional language hints. This is a separate route from chat, not a message part.<\/li>\n<li><strong>Audio in, reasoning out.<\/strong> Some models accept audio directly inside the messages array, so the model can summarise a meeting or judge tone without a separate transcription step.<\/li>\n<li><strong>Text in, audio out (speech synthesis).<\/strong> A text-to-speech endpoint returns audio bytes. Treat it as a separate service with its own latency profile.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Video is not a first-class modality in most APIs. In practice you sample frames into images and send them as multiple image parts, which means every video cost decision is really a frame-rate decision.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The request shape is still a chat completion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The single most important thing to internalise: the wire format barely changed. Instead of <code>\"content\": \"some text\"<\/code>, you send <code>\"content\": [ ... ]<\/code> \u2014 an ordered list of typed parts. Text parts and image parts can be interleaved, and order matters because it is the order the model reads them in.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from openai import OpenAI\nimport base64\n\nclient = OpenAI(\n    api_key=\"YOUR_API_KEY\",\n    base_url=\"https:\/\/your-gateway.example\/v1\",  # any OpenAI-compatible endpoint\n)\n\nwith open(\"invoice.jpg\", \"rb\") as f:\n    b64 = base64.b64encode(f.read()).decode(\"utf-8\")\n\nresp = client.chat.completions.create(\n    model=\"gpt-4o\",  # must be a vision-capable model on your gateway\n    messages=[\n        {\n            \"role\": \"system\",\n            \"content\": \"You extract fields from documents. Never guess: use null.\",\n        },\n        {\n            \"role\": \"user\",\n            \"content\": [\n                {\"type\": \"text\", \"text\": \"Return vendor, invoice date, currency and total.\"},\n                {\n                    \"type\": \"image_url\",\n                    \"image_url\": {\"url\": f\"data:image\/jpeg;base64,{b64}\"},\n                },\n            ],\n        },\n    ],\n    max_tokens=500,\n)\n\nprint(resp.choices[0].message.content)\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two details trip people up. First, the system prompt still governs behaviour \u2014 a vision request is not exempt from prompt design. Second, the model name must actually support image input. Sending an image part to a text-only model is a hard 400 error on most gateways, not a silent downgrade, which is a good thing: it fails loudly instead of quietly ignoring your image.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Modality cheat sheet<\/h2>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Modality<\/th><th>How you send it<\/th><th>What comes back<\/th><th>Main gotcha<\/th><\/tr><\/thead><tbody><tr><td>Image (photo, screenshot, chart)<\/td><td><code>image_url<\/code> part with a public URL or a <code>data:<\/code> base64 URI<\/td><td>Text, JSON, or tool calls<\/td><td>The detail \/ resolution setting can multiply token cost several times over<\/td><\/tr><tr><td>Multi-page document<\/td><td>One image part per rendered page<\/td><td>Text or JSON per page<\/td><td>Cost scales linearly with page count \u2014 page 40 costs the same as page 1<\/td><\/tr><tr><td>Short audio clip<\/td><td>Transcription endpoint (multipart file upload)<\/td><td>Plain text, optionally timestamps<\/td><td>Format and sample rate must be accepted by the endpoint<\/td><\/tr><tr><td>Long audio<\/td><td>Same endpoint, chunked client-side<\/td><td>Text per chunk<\/td><td>You own the stitching and the timestamp offsets<\/td><\/tr><tr><td>Audio as a reasoning input<\/td><td><code>input_audio<\/code> part inside <code>messages<\/code><\/td><td>Text or JSON<\/td><td>Only some models support it; verify before you design around it<\/td><\/tr><tr><td>Speech output<\/td><td>Text-to-speech endpoint<\/td><td>Audio bytes<\/td><td>Different latency budget \u2014 never block a chat UI on it<\/td><\/tr><tr><td>Video<\/td><td>Sampled frames as multiple image parts<\/td><td>Text<\/td><td>Frame rate is your cost dial; 1 fps is usually plenty<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Image understanding: what it is genuinely good at<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Vision models are strongest where structure is visual rather than textual. In production, the highest-value workloads are consistent:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Documents with layout.<\/strong> Invoices, receipts, purchase orders, insurance forms. The model sees the table, not a flattened column of numbers, so it can tell a subtotal from a total.<\/li>\n<li><strong>Screenshots.<\/strong> Support tickets with a screenshot attached, error dialogs, dashboards, browser state. This is often the fastest way to give an agent eyes on a UI.<\/li>\n<li><strong>Charts and diagrams.<\/strong> Reading a trend off a line chart or extracting node labels from an architecture diagram.<\/li>\n<li><strong>Physical inspection.<\/strong> Damage assessment, product condition, shelf compliance, safety-equipment checks. The model acts as a first-pass triage that routes the hard cases to a human.<\/li>\n<li><strong>Handwriting and messy scans.<\/strong> Skewed, shadowed, or partially obscured text where classic OCR struggles.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Where vision is <em>not<\/em> the right tool: high-volume, clean, machine-printed documents in a single font where you need character-exact accuracy at the lowest unit cost. Classic OCR is cheaper and more deterministic there. A sensible architecture uses OCR for the bulk scan and routes only low-confidence pages to a vision model.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Task<\/th><th>Better choice<\/th><th>Why<\/th><\/tr><\/thead><tbody><tr><td>10,000 clean printed invoices<\/td><td>OCR + rules<\/td><td>Deterministic, cheapest per page<\/td><\/tr><tr><td>Invoices with varied vendor layouts<\/td><td>Vision model<\/td><td>Layout-aware, no template per vendor<\/td><\/tr><tr><td>Handwritten notes<\/td><td>Vision model<\/td><td>OCR accuracy collapses on handwriting<\/td><\/tr><tr><td>Screenshot triage<\/td><td>Vision model<\/td><td>Requires UI and error-state reasoning<\/td><\/tr><tr><td>Exact barcode or MRZ strings<\/td><td>OCR + checksum validation<\/td><td>Verifiable, not probabilistic<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Audio: two products that people confuse<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Audio splits cleanly in two, and choosing the wrong one is the most common multimodal design mistake.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Transcription<\/strong> is a conversion service. You upload a file, you get text back. It is cheap relative to reasoning models and is the right choice when all you need is a searchable, quotable transcript.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Audio understanding<\/strong> is a reasoning service. You pass audio into the model and ask a question \u2014 &#8220;did the customer sound frustrated?&#8221;, &#8220;what were the action items?&#8221;. Use it when the answer depends on tone or background sound, because a transcript throws all of that away.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Transcription: multipart upload, separate from chat\ncurl https:\/\/your-gateway.example\/v1\/audio\/transcriptions \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -F file=@meeting.mp3 \\\n  -F model=whisper-1 \\\n  -F response_format=verbose_json \\\n  -F \"timestamp_granularities[]=segment\"\n\n# Typical verbose_json response (trimmed)\n# {\n#   \"text\": \"Let's ship the beta on Friday...\",\n#   \"language\": \"english\",\n#   \"duration\": 412.7,\n#   \"segments\": [\n#     {\"id\": 0, \"start\": 0.0, \"end\": 4.2, \"text\": \"Let's ship the beta on Friday.\"},\n#     {\"id\": 1, \"start\": 4.2, \"end\": 9.8, \"text\": \"I'll own the migration notes.\"}\n#   ]\n# }\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Chunk long audio yourself \u2014 split on silence so you never cut a word in half, and keep a running offset so timestamps stay aligned with the original recording.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The token math for non-text inputs<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Prices change constantly, so think in ratios rather than numbers. The stable mental model is this: <strong>images and audio are billed by how much information you hand over<\/strong>, and both are easier to over-send than text.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A small, low-detail image \u2014 a thumbnail or a simple icon \u2014 costs roughly the same as a short paragraph of text.<\/li>\n<li>A full-resolution photo or a scanned page at high detail can cost as much as several thousand tokens of text \u2014 often the largest line item in a vision request.<\/li>\n<li>Audio is typically billed by duration rather than tokens. A minute of audio lands in the same order of magnitude as a few thousand text tokens.<\/li>\n<li>Because image cost is driven by resolution, <strong>downscaling is your highest-leverage optimisation<\/strong>. Token cost follows pixel count, not file size.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Lever<\/th><th>What to do<\/th><th>Typical effect<\/th><\/tr><\/thead><tbody><tr><td>Resolution<\/td><td>Downscale to the smallest size where the answer is still correct<\/td><td>Largest single saving; often several times cheaper<\/td><\/tr><tr><td>Cropping<\/td><td>Send only the region of interest, not the whole screenshot<\/td><td>Proportional to area removed<\/td><\/tr><tr><td>Detail setting<\/td><td>Use low detail for classification, high detail only for fine text<\/td><td>Large, and easy to A\/B test<\/td><\/tr><tr><td>Frame rate (video)<\/td><td>Sample 1 frame per second instead of every frame<\/td><td>Linear in frame count<\/td><\/tr><tr><td>Caching<\/td><td>Hash the image and reuse the extracted result<\/td><td>100% saved on repeats<\/td><\/tr><tr><td>Two-stage routing<\/td><td>Cheap model triages, expensive model handles only hard cases<\/td><td>Large on skewed workloads<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">There is a second, less obvious cost: <strong>retries<\/strong>. A malformed response you re-request at full resolution doubles the bill for that image. Constrain the output before you optimise the input \u2014 see our guide to <a href=\"https:\/\/qoraapi.com\/blog\/ai-structured-outputs-json-mode\/\">structured outputs and JSON mode<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Combining vision with tool calls and structured output<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The real power shows up when modalities and capabilities compose. A vision model that can fill a schema and then call your tools is a product; one that only describes an image is a demo.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A pattern that works well for document intake:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Step 1 \u2014 Extract.<\/strong> Send the image with a JSON schema. The model returns typed fields, not prose. Nulls are explicit, so you can tell &#8220;not present&#8221; from &#8220;model missed it&#8221;.<\/li>\n<li><strong>Step 2 \u2014 Validate.<\/strong> Run your own checks: does the total equal the line items? Is the date plausible? Is the currency code in your allow-list?<\/li>\n<li><strong>Step 3 \u2014 Act.<\/strong> If validation passes, hand the object to a tool call that writes to your system of record. If it fails, route to a cheap vision model for a second read, then to a human queue.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This is where <a href=\"https:\/\/qoraapi.com\/blog\/ai-function-calling-tool-use\/\">function calling and the tool-use loop<\/a> becomes essential: the model decides <em>which<\/em> downstream action to take based on what it saw in the image, and your code stays in control of what is actually allowed to execute. The model never touches your database directly; it proposes a call, you authorise it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Production patterns that hold up<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Always pin the capability, not the model name.<\/strong> Keep a registry that maps a logical task (&#8220;document extraction&#8221;) to a list of models that support image input, and fail over down that list.<\/li>\n<li><strong>Pre-process before you send.<\/strong> Auto-rotate, downscale, and convert to a sane format client-side. You pay for pixels, and orientation metadata is free to fix.<\/li>\n<li><strong>Set <code>max_tokens<\/code> deliberately.<\/strong> Multimodal prompts invite rambling descriptions. Cap the output and ask for the shape you want.<\/li>\n<li><strong>Log the hash, not the image.<\/strong> Store a content hash plus the extracted result so you can prove idempotency and cache safely without hoarding user media.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Routing multimodal traffic<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Not every model in your catalogue accepts images, and fewer accept audio. That makes capability a <em>filter<\/em> that runs before your usual cost and quality routing. The correct order is: filter by modality support, then filter by context window, then pick the cheapest model that clears your quality bar.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Getting this wrong produces a specific failure mode: a request that works in staging, then 400s in production after a routing change sends it to a text-only tier. Encode the constraint in your router rather than in your prompt. Our <a href=\"https:\/\/qoraapi.com\/blog\/choose-right-ai-model-routing\/\">model-routing guide<\/a> covers that decision layer \u2014 the same four dimensions apply, with modality as a hard gate in front of them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Common mistakes<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Assuming OCR accuracy from a vision model.<\/strong> Vision models read context brilliantly and characters imperfectly. For serial numbers and amounts, validate against a checksum or a second read.<\/li>\n<li><strong>Uploading the original 12-megapixel photo.<\/strong> You are paying for detail the model will downsample away internally anyway.<\/li>\n<li><strong>Asking for a narrative when you need a field.<\/strong> &#8220;Describe this invoice&#8221; wastes tokens and invites hallucination. Ask for the four fields you will actually use.<\/li>\n<li><strong>Transcribing when you need understanding.<\/strong> If tone matters, a transcript is the wrong artefact.<\/li>\n<li><strong>Forgetting that user media is sensitive.<\/strong> Images and voice recordings are personal data. Hash, minimise, and set a retention window.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Getting started without a rewrite<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Because multimodal requests reuse the chat-completions shape, you do not need a second integration. If you route through an OpenAI-compatible relay such as <a href=\"https:\/\/qoraapi.com\/\" target=\"_blank\" rel=\"noopener\">qoraapi.com<\/a>, the same key and base URL serve vision, transcription, and text-only models \u2014 so switching the model behind a task is a one-line config change.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Start narrow: pick one high-volume document type, one vision model, and a schema. Measure accuracy on 100 real samples before optimising a single token.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What is a multimodal AI API?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">It is an API that accepts more than text as input. In practice that means you can attach images and audio to a request \u2014 usually as typed parts inside the same chat-completions message format \u2014 and receive text, structured JSON, or tool calls back. The advantage is one integration covering many input types.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Can I pass images as URLs instead of base64?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Usually yes. A public HTTPS URL is the cleaner option for images your storage already serves, because it keeps the request body small. Base64 is the safer option for private or user-uploaded media that is not publicly reachable. Some gateways only fetch URLs from allow-listed domains, so check before designing around remote URLs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How much more expensive is an image than text?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">It depends almost entirely on resolution and the detail setting. A small low-detail image costs about the same as a short paragraph; a full-resolution scan can cost as much as several thousand tokens of text. Downscaling and cropping usually cut image cost by a large multiple without changing answer quality.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Can I get structured JSON back from an image?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Yes, and you should. Combine the image part with a schema-constrained response format so the model returns typed fields rather than prose. That makes downstream validation trivial and dramatically reduces the retry rate \u2014 which matters more than the per-token price once images are in the request.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Is transcription part of the chat endpoint?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. Transcription is a separate route that takes a multipart file upload and returns text. Some models also accept audio inside the messages array for reasoning over a recording, but that is a distinct capability. Check which of the two your model supports before designing the flow.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Do I need a different SDK for multimodal requests?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Normally no. Mainstream SDKs already support typed content parts, so you keep the client you have and change the payload. That is the main reason to prefer OpenAI-compatible APIs: a multimodal upgrade becomes a payload change, not a platform migration.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The short version<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Multimodal is not a new API, it is a richer payload. Constrain the output, downscale the input, route by capability, and cache by content hash \u2014 those four habits separate a working prototype from a multimodal feature you can afford to run at scale.<\/p>\n\n\n\n\n<h3 class=\"wp-block-heading\">Related reading<\/h3>\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/qoraapi.com\/blog\/voice-ai-apis\/\">Building Voice AI Apps: TTS, STT, and Realtime APIs<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/image-generation-api-production\/\">Image Generation APIs in Production: Moderation, Caching, and Cost<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/document-data-extraction\/\">Extracting Structured Data from Documents with AI APIs<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-api-streaming-sse\/\">AI API Streaming Explained: How SSE Works and How to Consume It<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/prompt-management-versioning\/\">Prompt Management and Versioning in Production<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-mobile-integration\/\">Integrating AI APIs into Mobile Apps<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/multi-agent-orchestration\/\">Multi-Agent Orchestration: Patterns and Pitfalls<\/a><\/li><\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>How multimodal AI APIs work: send images and audio to models, control token cost, and combine vision with structured outputs and tool calls.<\/p>\n","protected":false},"author":1,"featured_media":95,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,6,9,7],"class_list":["post-96","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-api","tag-ai-api","tag-api-gateway","tag-developer-tools","tag-developers"],"_links":{"self":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/96","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/comments?post=96"}],"version-history":[{"count":2,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/96\/revisions"}],"predecessor-version":[{"id":266,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/96\/revisions\/266"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media\/95"}],"wp:attachment":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media?parent=96"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/categories?post=96"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/tags?post=96"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}