Running AI API Calls in Serverless and Edge

Serverless fits a subset of LLM workloads and fails the rest, and the deciding variable is almost always wall-clock time. Classification, embedding, and short single-shot completions that finish in under ten seconds are cheaper and simpler on a FaaS platform than on a container. Long generations, agent loops, and extended-reasoning calls are not: the platform kills them mid-response, and you pay for tokens the client never received. Decide per endpoint, not per architecture.

The mismatch is structural

The serverless contract assumes work that is short, bursty, stateless, and killable at any instant. LLM calls are long-tailed, incremental, and stateful across turns. Each mismatch maps to a constraint you cannot negotiate away:

Execution time limits. A hard platform ceiling, with no graceful shutdown.
Response size limits. API Gateway caps payloads at 10 MB; Lambda response streaming at 20 MB.
No durable local state. /tmp and in-process memory are recycled with the environment.
Cold starts. A fresh environment pays runtime init, dependency loading, and a TLS handshake first.
No guaranteed connection reuse. Pooling happens only if the platform reuses the same warm environment.
At-least-once invocation. Queues and client retries can run your handler twice.

What statelessness costs a chat product

Multi-turn chat needs history, which makes the function stateful by proxy. Pushing it to Redis or Postgres adds a round trip to every turn and turns a pure compute tier into a compute-plus-datastore tier. If that store lives in a VPC, the function’s network path and cold start are coupled to it too. Worse, prompt-prefix caching depends on an exact prefix, so a reordered history invalidates the cache you were counting on.

Platform constraints, compared

Three families matter, and the differences that decide the design are duration, streaming, and connection reuse across requests.

Property	Classic FaaS (AWS Lambda-style)	Edge runtime (Cloudflare Workers-style)	Container platform (Fargate, Cloud Run, Fly.io)
Max duration	15 minutes per invocation; 29 seconds if fronted by API Gateway	No fixed wall-clock limit while a response stream is open; CPU time is capped separately, commonly 30 seconds default and extendable	Cloud Run defaults to 300 seconds per request, configurable to 60 minutes; long-running services have no per-request cap
Streaming support	Only via Function URLs with response streaming; never through API Gateway	Native and first-class; the runtime is built around `ReadableStream`	Native; the only family that streams reliably through a normal load balancer
Outbound connection reuse	Within a warm execution environment only, and only if the client lives outside the handler	Within an isolate, best-effort; isolates are recycled aggressively	Real keep-alive pools that survive across requests, with tunable pool sizes
Cold start profile	Hundreds of milliseconds to several seconds, driven by runtime, bundle size, and VPC configuration	Low single-digit milliseconds for isolate creation; the real cost is the first outbound TLS handshake in a new isolate	Seconds from zero, but eliminated by keeping a minimum instance count
Runtime limits	Node, Python, Java, .NET, Go, Rust; 128 MB to 10 GB memory; 50 MB zipped deployment package; no inbound sockets	V8 isolates with Web Platform APIs; 128 MB memory; no filesystem; subrequest counts capped per invocation	Whatever fits in a container; no meaningful runtime restrictions
Best fit	Bursty, short, spiky calls: classification, embedding, single-shot completions	Latency-sensitive short calls and streaming proxies where compute is mostly waiting	Sustained concurrency, long generations, agent loops, in-process prompt caches

The column that quietly decides most architectures is outbound connection reuse. A container keeps a TLS session to the provider open for hours; a function keeps one only while its environment survives, and an edge isolate only until eviction. Duration, memory, and runtime can be worked around. That one cannot, which is why serverless LLM deployments so often route through a long-lived intermediary.

Execution time limits are the design constraint

Treat the platform ceiling as the primary filter. A workable rule of thumb: compare your p95 end-to-end provider latency against roughly a quarter of the platform’s wall-clock budget, because you need headroom for cold start, retries, and the tail. If the budget is 30 seconds, do not inline anything with a p95 above seven or eight seconds. At 15 minutes, the threshold moves to three or four minutes.

Workloads that fit

Short completions. Extraction, rewriting, labelling, short structured output.
Classification and routing. Low single-digit latency, deterministic token counts.
Embeddings. No streaming, bounded output, per-item billing. The cleanest fit here.
Small RAG answers. Retrieve, stuff a bounded context, generate briefly — if retrieval is not a slow VPC hop.

Workloads that do not fit

Long generations. 4,000 tokens at 40 per second is 100 seconds of streaming: fine on an edge runtime, fatal behind a 29-second integration timeout.
Deep reasoning. Time-to-first-token alone can exceed a FaaS front door’s entire budget.
Batch jobs. Per-item overhead dominates; these belong on workers that amortise startup.
Agent loops. A ten-step loop of tool calls multiplies the odds of crossing the ceiling.

A well-behaved serverless LLM handler builds its HTTP client at module scope so the pool survives across warm invocations, sets timeouts per phase, and records when it exceeded its own budget — that log line is the leading signal that the workload is outgrowing the platform.

import os
import time
import httpx

# Module scope. A client created inside the handler pays a fresh TCP and
# TLS handshake on every single invocation.
_client = httpx.Client(
    base_url="https://api.qoraapi.com",
    timeout=httpx.Timeout(connect=2.0, read=8.0, write=2.0, pool=1.0),
    limits=httpx.Limits(max_keepalive_connections=20, max_connections=50),
    headers={"authorization": "Bearer " + os.environ["QORA_API_KEY"]},
)

BUDGET_S = 8.0  # the p99 we are willing to absorb inside the function

def classify(text: str) -> str:
    started = time.monotonic()
    resp = _client.post(
        "/v1/chat/completions",
        json={
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "system", "content": "Reply with exactly one label."},
                {"role": "user", "content": text},
            ],
            "max_tokens": 8,
            "temperature": 0,
        },
    )
    resp.raise_for_status()
    elapsed = time.monotonic() - started
    if elapsed > BUDGET_S:
        # Succeeded, but only because the platform had headroom left.
        # Emit this as a metric: it is the leading indicator of a bad fit.
        print("over-budget call: %.2fs" % elapsed)
    return resp.json()["choices"][0]["message"]["content"].strip()

Streaming from serverless

Streaming exists to move time-to-first-token from seconds to hundreds of milliseconds. Buffering the whole response inside the function destroys that property and adds a failure mode: the entire output must fit in memory and inside the payload limit. If you are going to buffer, you have chosen a synchronous API and should use the job-and-webhook pattern instead.

Passing a stream through a function

The mechanics differ by platform but the shape does not: read the upstream body as a stream, write each chunk to the response stream as it arrives, never accumulate. On Lambda that means response streaming through a Function URL using the streaming handler signature — API Gateway cannot do it, and its payload limit and 29-second integration timeout make the question moot. On edge runtimes it is the default: return a ReadableStream and the runtime forwards chunks as produced.

The heartbeat problem

Every hop between your function and the client has an idle timer, and all of them are shorter than a slow generation. An ALB idles a connection out at 60 seconds by default. nginx’s proxy_read_timeout also defaults to 60. The timer measures inactivity, not total duration, so the fix is to ensure no gap ever approaches it: emit an SSE comment line every 15 seconds. Comment lines begin with a colon and are ignored by every conforming parser, so the cost is a few bytes and the benefit is surviving a model that pauses before its first token.

Two details bite in production. Buffering proxies must be disabled — send X-Accel-Buffering: no and Cache-Control: no-cache, no-transform, or an intermediary holds your chunks until it has enough to flush, reintroducing the latency you removed. And compression must be off; a gzip filter with a 1 KB buffer will sit on your first token until the second arrives.

Cold starts: what actually causes them

Cold start is not one delay. It is four, and they are separable.

Runtime initialisation. Node and Python are fast; a JVM is not.
Dependency loading. A fat SDK that drags in half of AWS adds hundreds of milliseconds.
TLS handshake to the provider. A client built per invocation pays a full handshake before the request is sent.
DNS resolution. Often ignored, occasionally dominant — especially in a VPC, where queries hit the VPC resolver.

Mitigations that work

Smaller bundles. Ship the HTTP client you need, not a framework. Costs nothing at runtime.
Module-scope clients. Build them outside the handler so they survive between invocations.
Provisioned concurrency. Environments already past runtime init and holding an open provider connection. The only fix that removes the handshake cost rather than hiding it.
More memory, on Lambda. CPU scales with memory, so this genuinely shortens CPU-bound init.

What is cargo cult

A scheduled ping every five minutes to keep the function warm does not reliably work. It warms one execution environment, it does not control which environment serves the next real request, and under concurrency you need as many warm environments as simultaneous requests. Warm-up plugins on a schedule share the flaw. The old claim that VPC configuration adds ten seconds to every cold start is obsolete — hyperplane network interfaces fixed it — though VPC still adds latency through NAT hops and DNS.

resource "aws_lambda_function" "llm_proxy" {
  function_name = "llm-proxy"
  runtime       = "nodejs20.x"
  handler       = "index.handler"

  # CPU scales with memory, so this also shortens CPU-bound cold start.
  memory_size = 1024
  # A hard ceiling. Exceeding it terminates the invocation mid-write.
  timeout = 30

  filename         = data.archive_file.bundle.output_path
  source_code_hash = data.archive_file.bundle.output_base64sha256

  environment {
    variables = {
      # An ARN, not a key. The function fetches and caches the secret itself.
      QORA_API_KEY_SECRET_ARN = aws_secretsmanager_secret.qora.arn
      UPSTREAM_TIMEOUT_MS     = "20000"
    }
  }

  # Pre-initialised environments: past runtime init and connection-warm.
  # Billed whether or not they serve traffic.
  provisioned_concurrency_config {
    provisioned_concurrent_executions = 5
  }
}

resource "aws_lambda_alias" "live" {
  name             = "live"
  function_name    = aws_lambda_function.llm_proxy.function_name
  function_version = "$LATEST"
}

# 29000 is the maximum. API Gateway will not let you raise it.
resource "aws_api_gateway_integration" "proxy" {
  http_method             = "POST"
  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  timeout_milliseconds    = 29000
  uri                     = aws_lambda_function.llm_proxy.invoke_arn
}

Connection reuse and TLS

A TLS handshake to a provider endpoint costs roughly two network round trips: TCP plus TLS negotiation, with TLS 1.3 getting a resumed session down to one. Call it 40 milliseconds to a region a few hundred kilometres away. Against a 20-second generation that is noise. Against a 300-millisecond classification call it is 13 percent of your latency, and if the function also pays runtime init and dependency loading, it can be the second-largest component of the request.

The arithmetic inverts the usual intuition: the shorter your LLM call, the more a cold handshake matters proportionally — which is exactly the workload serverless is supposed to be good at. The fix takes two forms. Where the platform allows it, keep the client at module scope so warm environments reuse the socket; that is free and you should always do it. Where it does not, route through a gateway holding a pooled upstream connection, so your function talks to one nearby endpoint instead of handshaking with a distant provider region. That also removes per-provider pools from your code, and is a large part of why teams put a gateway in front of their providers rather than calling them directly.

Secrets: environment variables are a liability

A provider key in a function’s environment variables is readable by anyone holding lambda:GetFunctionConfiguration, stored in plaintext in your Terraform state, leaked into crash dumps and error payloads, and impossible to rotate without redeploying every function that references it. With forty functions across four environments that is a coordinated fleet-wide deploy, which in practice means the key never rotates.

Secret manager with a cached fetch. Store the ARN, fetch on first use, cache in module scope, refresh on a TTL.
Short-lived credentials. Use the platform identity — IAM role, workload identity, OIDC — to mint a short-lived token.
One scoped key to a gateway. The function holds a single revocable key; the gateway holds the provider credentials. Rotation is one change in one place.

This is the same reasoning behind any sound LLM API security posture: minimise long-lived secrets, and make the ones that remain cheap to rotate.

The cost model: where the crossover is

Serverless wins on spiky traffic and loses on sustained traffic, and the crossover is calculable. Take published on-demand rates: Lambda at $0.0000166667 per GB-second plus $0.20 per million requests, Fargate at $0.04048 per vCPU-hour plus $0.004445 per GB-hour.

Worked example. Each request runs 4 seconds at 512 MB, so it consumes 4 x 0.5 = 2 GB-seconds. Per-request cost is 2 x $0.0000166667 = $0.00003333, plus $0.0000002 for the request itself, totalling $0.00003353.

Spiky, 50,000 requests per day. Lambda: 50,000 x $0.00003353 = $1.68/day, about $50/month. Peak load is near 14 concurrent calls, so containers need two 1 vCPU / 2 GB tasks: 2 x ($0.04048 + 2 x $0.004445) x 730 = $72, plus ~$20 for an ALB. Roughly $92. Serverless wins.
Steady, 1,000,000 requests per day. Lambda: 1,000,000 x $0.00003353 = $33.53/day, about $1,006/month. That averages 11.6 requests per second, roughly 46 concurrent calls; at a conservative 15 per 1 vCPU task, four tasks suffice: 4 x $36.04 = $144, plus $25 for an ALB. Roughly $170. Containers win by about six times.

The crossover sits where Lambda’s monthly bill equals the container floor of about $170: roughly 5.1 million requests per month, or 170,000 per day — a sustained two requests per second. Below that, serverless is cheaper. Above it, you pay a per-invocation premium for elasticity you no longer need.

Profile	Requests per day	Serverless monthly	Container monthly	Cheaper option
Spiky, 4-hour peak window	50,000	~$50	~$92	Serverless
Sustained, flat	1,000,000	~$1,006	~$170	Container, by ~6x
Crossover	~170,000	~$170	~$170	Indifferent on cost; decide on latency and control

Two caveats shift the crossover toward containers. Provisioned concurrency is billed while idle, adding to the serverless column without adding throughput. And if your handler is CPU-bound rather than waiting on the network, the concurrent calls a container task can serve drops sharply.

Failure modes specific to serverless

Silent truncation mid-stream

Once the first SSE chunk is flushed, the HTTP status is committed. If the function then hits its timeout you cannot send a 500 — the client already has a 200 and a partial body. It sees a stream that simply stops, indistinguishable from a network blip, and your application may store a half-finished answer as complete. The fix is an in-band terminal marker: emit an explicit event: truncated frame when the read loop exits without a [DONE] or finish_reason, and make clients treat a stream that closes without one as a failure regardless of status code.

Duplicate execution on retry

Async invocations retry on failure, queues redeliver after a visibility timeout, and API clients retry on connection errors. A retried LLM call is a second billable generation and, if it writes to your database, a duplicated record. You cannot make the provider call idempotent, so idempotency has to live at your boundary: an idempotency key, a short-lived record of completed work, and a client contract that treats a repeat as a lookup. This is the discipline that makes asynchronous job delivery safe, and it applies equally to a synchronous endpoint that retries.

Losing the response when the client disconnects

When the browser tab closes, the function does not necessarily stop. On some runtimes the invocation runs to completion and you pay for every token. On others the runtime aborts the handler, and whether that abort reaches the in-flight upstream request depends entirely on your code. Propagate it explicitly: listen on the request’s abort signal and cancel the upstream fetch, so a disconnected client stops the generation instead of billing you for output nobody reads.

VPC and networking surprises

Placing a function in a VPC to reach a database adds a NAT hop to every outbound provider call: latency plus a per-gigabyte data processing charge. It also changes DNS resolution, so the first lookup in a cold environment can be slower than expected. If the function only needs the public internet and a database with a public endpoint, do not put it in a VPC. If it must, budget the NAT cost into the arithmetic above.

A streaming LLM endpoint on an edge runtime

This handler covers the two cases that break naive implementations: the deadline, and the client walking away. It never buffers, heartbeats on an interval, distinguishes an error before the first byte (where a real status code is still available) from one after (where only an in-band frame is), and aborts upstream on disconnect.

interface Env {
  QORA_API_KEY: string;
}

const UPSTREAM = "https://api.qoraapi.com/v1/chat/completions";
const DEADLINE_MS = 90_000;   // our own budget, below the platform ceiling
const HEARTBEAT_MS = 15_000;  // comfortably under a 60s proxy idle timeout

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== "POST") {
      return new Response("method not allowed", { status: 405 });
    }

    const payload = await request.text();
    const upstreamAbort = new AbortController();

    // Client disconnect must cancel the upstream generation, not just the response.
    request.signal.addEventListener("abort", () => {
      upstreamAbort.abort("client-disconnect");
    });

    const deadline = setTimeout(() => {
      upstreamAbort.abort("deadline-exceeded");
    }, DEADLINE_MS);

    let upstream: Response;
    try {
      upstream = await fetch(UPSTREAM, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: "Bearer " + env.QORA_API_KEY,
        },
        body: payload,
        signal: upstreamAbort.signal,
      });
    } catch (err) {
      clearTimeout(deadline);
      // Nothing has been flushed yet, so a real status code is still available.
      return new Response(JSON.stringify({ error: String(err) }), {
        status: 504,
        headers: { "content-type": "application/json" },
      });
    }

    if (!upstream.ok || upstream.body === null) {
      clearTimeout(deadline);
      return new Response(upstream.body, { status: upstream.status });
    }

    const encoder = new TextEncoder();
    const decoder = new TextDecoder();

    const stream = new ReadableStream<Uint8Array>({
      async start(controller) {
        const reader = upstream.body!.getReader();
        const heartbeat = setInterval(() => {
          // SSE comment line: ignored by parsers, resets every idle timer.
          controller.enqueue(encoder.encode(": keep-alive\n\n"));
        }, HEARTBEAT_MS);

        let tail = "";
        let sawTerminal = false;

        try {
          for (;;) {
            const { done, value } = await reader.read();
            if (done) break;
            tail = (tail + decoder.decode(value, { stream: true })).slice(-256);
            controller.enqueue(value);
            if (tail.includes("[DONE]") || tail.includes('"finish_reason"')) {
              sawTerminal = true;
            }
          }
        } catch (err) {
          // Status is already committed. The only channel left is in-band.
          const frame = JSON.stringify({ message: String(err) });
          controller.enqueue(encoder.encode("event: error\ndata: " + frame + "\n\n"));
        } finally {
          clearInterval(heartbeat);
          clearTimeout(deadline);
          if (!sawTerminal) {
            controller.enqueue(
              encoder.encode(
                'event: truncated\ndata: {"reason":"upstream ended without a terminal frame"}\n\n'
              )
            );
          }
          controller.close();
        }
      },

      cancel(reason) {
        // Response stream cancelled downstream: stop paying for the generation.
        upstreamAbort.abort(String(reason));
      },
    });

    return new Response(stream, {
      headers: {
        "content-type": "text/event-stream; charset=utf-8",
        "cache-control": "no-cache, no-transform",
        "x-accel-buffering": "no",
      },
    });
  },
};

Two things this handler deliberately does not do. It does not retry the upstream call, because a retry after partial output would append a second generation to the first; retries belong before the first byte or in a separate job. And it does not resolve a model to a provider — that is the gateway’s job, and keeping it there means failover across providers happens without redeploying the function.

When not to use serverless, and where I would default

Five conditions disqualify it, and any one of them is sufficient.

Long generations. If p95 time-to-last-token exceeds a quarter of the ceiling, you are one bad tail latency from truncation.
Sustained high concurrency. Past two requests per second you pay an elasticity premium for capacity you could buy outright.
Heavy in-process state. A prompt cache, embedding index, or reranker — anything whose value depends on state surviving between requests.
GPU needs. Cold-starting a GPU costs more than the request you were serving.
A durable provider connection. A long-lived multiplexed connection or websocket upstream rules serverless out.

So: default to serverless for embedding, classification, extraction, short single-shot completions, and the generation half of a small RAG pipeline — anything with a p95 under about eight seconds and bounded output. You get near-zero fixed cost, trivial scaling, and no capacity planning, and the cold-start TLS penalty is manageable with a module-scope client. Default to containers for anything conversational at scale, anything streaming more than a few hundred tokens, anything with an agent loop, and anything where a local cache changes your cost profile. The moment you configure provisioned concurrency to hit a latency SLO, you are paying the container floor while still accepting serverless constraints. Time to move.

The hybrid that actually ships is a thin serverless or edge tier in front of a long-lived worker tier, with a gateway between them holding upstream credentials and pooled connections. The front tier absorbs bursts and terminates client connections; the worker tier owns the slow, stateful, expensive work.

Frequently asked questions

Can I just raise the function timeout?

Up to the platform maximum, yes, and it is often the right first move — Lambda allows 15 minutes per invocation. What you cannot raise is the timeout of the layer in front of it. API Gateway’s 29-second integration timeout is hard, an ALB’s 60-second idle timeout is configurable but bounded, and Cloudflare returns a 524 if an origin does not respond in time. Raising the function timeout only helps if the path bypasses those layers, for example a Function URL with response streaming. Measure the smallest ceiling in the path, not the largest.

Does provisioned concurrency solve cold starts?

It solves the parts you can pre-warm: runtime init, dependency loading, and the TLS handshake if your client lives at module scope. It does not solve anything downstream — provider-side queueing, model warm-up, or your database’s connection establishment. It is also billed whether or not it serves traffic, which makes it the clearest signal that a workload has outgrown serverless.

Can I stream from AWS Lambda?

Yes, but only through a Function URL with response streaming, using the streaming handler signature rather than the standard one. API Gateway cannot stream responses, and its 10 MB payload limit applies to the buffered case. Even with response streaming, plan for the 20 MB soft response limit and for the fact that once you write the first chunk your status code is fixed, so mid-stream failures must be signalled in-band. If streaming is central to your product, an edge runtime fits better than Lambda.

Should I put my provider API key in the function environment?

For a prototype, fine. Past a couple of functions, no. Environment variables are readable by anyone with configuration-read permission, stored in plaintext in your infrastructure state, visible in error payloads, and require a fleet-wide redeploy to rotate. Store the secret’s ARN instead and fetch it once per execution environment with a TTL cache, or issue a single scoped key pointing at a gateway that holds the real provider credentials.

Conclusion

Serverless and edge functions are a good home for short, bounded, stateless LLM calls and a bad home for long, streaming, stateful ones. The decision reduces to wall-clock time against the smallest ceiling in your request path, and it is worth making per endpoint rather than per system. Where the workload fits: keep the HTTP client at module scope, heartbeat your streams, signal truncation in-band, make retries idempotent, and never put a long-lived provider key in an environment variable. Where it does not: move to a container and stop paying for elasticity you are not using. For the middle ground — bursty front ends, pooled upstream connections, one place to rotate credentials — a gateway such as QoraAPI lets a short-lived function behave like a long-lived client.

Running AI API Calls in Serverless and Edge Functions