Blog

The real cost of running AI agents in production

Apr 15, 2026

Chatbots are cheap. Agents are not.

A chatbot sends a user message, gets a response, displays it. Maybe 2,000 tokens per exchange. An agent reads files, calls tools, retries on errors, re-sends the entire conversation every step, and does this 20–60 times per task. Same API, completely different economics.

If you’re budgeting for AI agents the same way you budget for a chatbot, you’re underestimating by 10–50x.

Token consumption: chatbot vs. agent

We measured token consumption across three workload types, each running for one hour:

Coding agent (OpenClaw)

~2.1M tokens

Research agent (CrewAI)

~1.2M tokens

RAG chatbot

~200K tokens

Simple chatbot

~40K tokens

The coding agent consumed 52x more tokens than a simple chatbot in the same time period. And this is normal — the agent was doing useful work the entire time.

Why agents cost so much

Three architectural properties of agents make them expensive:

1. Context accumulation

Every agent step appends tool outputs to the conversation. The LLM re-processes the entire conversation on each step. If the agent reads a 3,000-token file at step 5, that file gets re-sent at steps 6, 7, 8… all the way to the end.

For a 40-step task, one file read costs: 3,000 tokens × 35 remaining steps = 105,000 tokens in re-transmission.

This is why agent token consumption grows quadratically, not linearly.

2. System prompt overhead

Agent frameworks use large system prompts — OpenClaw’s is ~9,600 tokens, CrewAI’s varies by agent configuration. This prompt is sent with every request. Over 40 steps, the system prompt alone costs 384,000 tokens.

3. Error retry loops

When a tool call fails, the agent retries. Each retry sends the full context plus the error message. Three retries on a 30K-token context wastes 90K tokens with no productive output.

Without a retry cap, this can run indefinitely — always bound agents with a retry cap and a maximum iteration count.

Monthly cost by model and framework

Assuming one developer running 15 agent tasks per day, 22 working days per month, ~500K tokens per task (80% input / 20% output, at each vendor’s list price — live prices here):

Model	Cost/task	Daily (×15)	Monthly
Claude Fable 5	$9.00	$135.00	$2,970
GPT-5.5	$5.00	$75.00	$1,650
Claude Opus 4.8	$4.50	$67.50	$1,485
GLM 5.2 (open)	$1.00	$15.00	$330
DeepSeek V4 Flash (open)	$0.08	$1.26	$28
CheapestInference (full day)	—	—	from $48.45/mo flat

A team of 5 developers each running 15 tasks/day on Claude Fable 5 spends $14,850/month. The same team on flat-rate via CheapestInference pays a fixed monthly subscription per seat (from $48.45/mo for a reserved daily time block) — no matter how many tokens those agents burn. That’s an order-of-magnitude reduction.

Four strategies to cut agent inference costs

1. Switch to open-source models

Open-weight models like Kimi K2.6 and MiniMax M3 now sit at parity with Gemini 3.1 Pro on SWE-bench Verified, and GLM 5.2 outscores GPT-5.5 on SWE-bench Pro — at a half to a sixth of the per-token price. Full data: the Which-LLM guide and the State of Open Weights report.

2. Route by task complexity

Not every agent step needs a frontier model. File reads, simple classifications, and formatting don’t need 685B parameters. Use a small model for easy steps and a large model for hard ones. Full guide: Building a multi-model architecture.

3. Give each agent its own key

Give each agent its own API key so one runaway agent can’t starve the others. On a time-block subscription each key gets unlimited usage during its reserved hours, so you isolate workloads without juggling per-token allocations.

4. Use flat-rate pricing

Per-token pricing penalizes the exact patterns agents use: large contexts, many steps, retries. Flat-rate pricing makes all of that free. During your reserved time blocks your agent can use the full context window and retry freely without increasing the bill — reserve all three blocks for 24/7 coverage.

The math that matters

Here’s the equation most teams miss:

Agent cost = tokens_per_step × steps × cost_per_token

Most optimization focuses on cost_per_token — switching to a cheaper model. But tokens_per_step grows with context (quadratic), and steps is unpredictable. Optimizing only one variable leaves the other two working against you.

Flat-rate pricing eliminates all three variables from your bill. The cost is the subscription. Period.

We serve six open-weight models across two flat-rate pools — Kimi K2.7, Kimi K2.6, GLM 5.2, and MiniMax M3 in the Frontier pool (from $48.45/mo), DeepSeek V4 Flash and MiMo v2.5 in the Core pool (from $12.74/mo) — with unlimited time-block subscriptions: no token counting, no budget caps during your reserved hours. Reserve 1–3 daily 8-hour blocks and your agent’s token consumption never becomes your problem. Get started or see plans.

OpenClaw is free. Running it is not.

Mar 24, 2026

OpenClaw has 247,000 GitHub stars. It’s free, open-source, and runs locally. You install it, point it at an LLM, and it writes code, browses the web, queries databases, and executes files on your behalf.

The agent is free. The inference is not.

Every time OpenClaw calls a model, it re-sends the entire conversation history — every tool output, every file it read, every intermediate result. By iteration 20 of a typical task, the input context is 30,000+ tokens. By iteration 40, it’s past 100,000. And it sends this every single request.

This is not a bug. It’s how agents work. And it’s why running OpenClaw on pay-per-token APIs costs $300–600/month for active users — sometimes more.

Where the tokens go

We broke down token consumption for a typical OpenClaw coding task: “add authentication to an Express API.” The agent completed it in 38 tool calls.

Context accumulation

~280K tokens

System prompt (×38)

~156K tokens

Tool outputs (files, etc.)

~70K tokens

Agent output

~19K tokens

Total: ~525,000 tokens for a single task. The agent’s actual output — the code it wrote — was 19K tokens. The other 96% is overhead.

On Claude Opus at $15/M input + $75/M output, that single task costs $9.18. Run five tasks a day and you’re at $1,377/month.

On DeepSeek V3.2 via a pay-per-token provider at $0.27/M input + $1.10/M output, the same task costs $0.16. Better — but 20 tasks a day is still $96/month, and that’s one agent.

The three cost traps

Here’s the OpenClaw-specific version:

1. Context grows quadratically

OpenClaw reads files into context. If it reads a 2,000-token file at step 5, that file gets re-sent at steps 6, 7, 8… all the way to 38. That single file read costs 2,000 × 33 remaining steps = 66,000 tokens in re-transmission alone.

Users report session contexts at 56–58% of the 400K context window during normal use. This isn’t a failure mode — it’s the architecture working as designed.

2. System prompt is a fixed tax

OpenClaw’s system prompt is ~9,600 tokens. It gets sent with every request. Over 38 tool calls, that’s 365K tokens just in system prompts. You pay this whether the agent does useful work or not.

3. Wrong model for the job

OpenClaw defaults to a single model for everything. But not every tool call needs the same intelligence:

Reading a file and deciding what to edit? Llama 3.1 8B handles this at 200 tokens/sec.
Writing complex authentication logic? A frontier open-weight model like Kimi K2.7 is the right call.
Formatting a config file? Any 8B model is overkill but still cheaper than Opus.

We wrote a full guide on this pattern: Building a multi-model architecture. Routing agent requests to the right model can cut costs by 60–80% without reducing output quality.

The math on flat-rate vs. pay-per-token

Here’s the comparison for an OpenClaw user running ~20 tasks/day:

Provider	Cost/task	20 tasks/day	Monthly
Claude Opus (direct)	$9.18	$183.60	$5,508
GPT-5.4 (direct)	$4.73	$94.60	$2,838
DeepSeek V3.2 (per-token)	$0.16	$3.20	$96
CheapestInference (flat-rate)	—	—	from $12.74/mo (Core) · $48.45/mo (Frontier)

Flat-rate means you don’t care about context accumulation. The 280K tokens of context overhead that makes pay-per-token expensive? Irrelevant. The system prompt tax? Doesn’t matter. Your agent can call models 24/7 and the bill is the same.

If you’re running OpenClaw, here’s the setup we see working best:

1. Use open-weight models. Frontier open-weight models like Kimi K2.7 and GLM 5.2 score within a few points of proprietary models on coding benchmarks (the data). The gap doesn’t justify a 50x cost difference.

2. Route by complexity. Don’t send file reads and simple decisions to the same model as complex code generation. A router model costs fractions of a cent per classification. Full guide: Multi-model architecture.

3. Reserve the hours you work. On CheapestInference you subscribe to a pool and reserve one or more daily 8-hour time blocks (Asia-Pacific, Europe, Americas — pick 1–3, all three is full 24/7). The Frontier pool (from $48.45/mo billed annually) carries Kimi K2.7, Kimi K2.6, GLM 5.2, and MiniMax M3; the Core pool (from $12.74/mo) carries DeepSeek V4 Flash and MiMo v2.5 — both 1M-context models that handle everyday OpenClaw tasks for the price of a coffee. During your reserved hours inference is unlimited with no budget cap. One API key per agent, one request at a time per key. Outside your window, requests return 429 until your block opens again.

4. Handle rate limits automatically. Time blocks mean your agent will hit 429s outside your reserved window — that’s expected. But OpenClaw kills the conversation when it gets a 429. The agent stops, and if you close the dashboard, that conversation is gone.

We built an OpenClaw plugin that fixes this: openclaw-ratelimit-retry. It hooks into agent_end, detects retriable 429s, parks the session on disk, and waits for the budget window to reset. Then it sends chat.send to the original session — resuming the conversation with its full transcript, as if you had typed a message.

openclaw plugins install @cheapestinference/openclaw-ratelimit-retry

plugins:
  ratelimit-retry:
    budgetWindowHours: 8    # matches your CheapestInference 8-hour time block
    maxRetryAttempts: 3     # give up after 3 consecutive 429s
    checkIntervalMinutes: 5 # check every 5 min for ready retries

The plugin is zero-dependency, persists across server restarts, deduplicates by session, and handles edge cases like sub-agents, queue overflow, and corrupted state files. If the retry itself hits a 429, it re-queues automatically. No tokens wasted on re-sending from scratch — the agent picks up exactly where it left off.

This turns budget caps from “your agent crashes” into “your agent naps and wakes up.” Set it up once and forget about it.

5. Consider unlimited time blocks. If your agent runs more than a few tasks per day, per-token pricing works against you. Every token of context overhead is money. With an unlimited time-block subscription, context overhead is free during your reserved hours — re-send the full window, let the agent work without a budget cap.

The irony

OpenClaw is free because the code runs on your machine. But the valuable part — the intelligence — runs on someone else’s GPUs. The agent framework is the cheap part. Inference is the expensive part.

Open-source models on flat-rate infrastructure flip this equation. The models are free. The inference is flat. The only variable cost left is your time.

Point your OpenClaw base_url at https://api.cheapestinference.com/v1 and find out what unconstrained agents actually cost: nothing more than you already budgeted.

Building a multi-model architecture: route requests to the right LLM

Mar 19, 2026

Using one model for everything is the simplest architecture. It’s also the most wasteful. A 685B-parameter reasoning model answering “what’s the weather?” is like hiring a PhD to sort mail.

This guide covers how to use a small, fast model to classify incoming requests and route them to the right specialist. The result: lower latency, lower cost, and often better quality — because each model handles what it’s actually good at.

The problem with single-model architectures

Most applications start with one model:

User request --> Large Model --> Response

This works, but every request — simple or complex — pays the same latency and cost penalty. When 60% of your traffic is simple classification, FAQ, or extraction, you’re burning expensive compute on tasks a small model handles equally well.

Llama 3.1 8B

~200 t/s

DeepSeek V3.2

~60 t/s

DeepSeek R1

~30 t/s

The gap between Llama 8B and R1 is nearly 7x in throughput. Routing simple requests to the small model saves that difference on every request.

The multi-model architecture

User request --> Router (GLM 5.2) --> classify intent
                                          |
                  +-----------+-----------+-----------+
                  |           |           |           |
           simple/general  reasoning    code        agent
                  |           |           |           |
               GLM 5.2    MiniMax M3  MiniMax M3  Kimi K2.7
                  |           |           |           |
                  +-----+-----+-----+-----+
                        |
                     Response

Two stages:

Classify — The router model reads the user’s message and outputs a category. A fast model returns this in a fraction of a second.
Route — Based on the category, forward the request to the appropriate specialist model.

The router adds minimal overhead (~200ms) but saves significant compute by keeping simple requests away from expensive models.

Step 1: Classify with a fast model

A fast, lightweight model makes a good router. With low TTFT and a short, single-word output, the classification step costs almost nothing and completes before the user notices. (On CheapestInference, DeepSeek V4 Flash or MiMo v2.5 in the Core pool are natural router models; the example below uses GLM 5.2 so everything runs on one Frontier subscription.)

The classification prompt is simple — you want a single-word category, not a conversation:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.cheapestinference.com/v1",
    api_key="your-api-key"
)

def classify_request(user_message: str) -> str:
    """Classify a user message into a routing category."""
    response = client.chat.completions.create(
        model="glm-5.2",
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify the user's message into exactly one category. "
                    "Respond with only the category name, nothing else.\n\n"
                    "Categories:\n"
                    "- simple: greetings, FAQ, simple factual questions\n"
                    "- general: complex questions, analysis, writing, summarization\n"
                    "- reasoning: math, logic, multi-step problems, science\n"
                    "- code: code generation, debugging, refactoring, technical implementation\n"
                    "- agent: tasks requiring tool use, web search, or multi-step execution"
                )
            },
            {"role": "user", "content": user_message}
        ],
        max_tokens=10,
        temperature=0
    )
    category = response.choices[0].message.content.strip().lower()
    # Default to general if classification is unclear
    valid = {"simple", "general", "reasoning", "code", "agent"}
    return category if category in valid else "general"

The key details: max_tokens=10 because we only need one word. temperature=0 for deterministic routing. The system prompt is explicit about format — no preamble, just the category.

Step 2: Route to the specialist

Each category maps to a model optimized for that task:

# Model routing table
ROUTE_TABLE = {
    "simple":    "glm-5.2",
    "general":   "glm-5.2",
    "reasoning": "MiniMax-M3",
    "code":      "MiniMax-M3",
    "agent":     "kimi-k2.7",
}

def route_request(user_message: str, conversation_history: list) -> str:
    """Classify and route a request to the appropriate model."""
    category = classify_request(user_message)
    model = ROUTE_TABLE[category]

    response = client.chat.completions.create(
        model=model,
        messages=conversation_history + [
            {"role": "user", "content": user_message}
        ],
        stream=True
    )

    # Stream the response back
    full_response = ""
    for chunk in response:
        if chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            full_response += content
            print(content, end="", flush=True)

    return full_response

Notice that simple requests route back to GLM 5.2 — the same model that did the classification. For simple queries, the router overhead is effectively zero because the specialist is the same model and can reuse the warm connection.

Step 3: Handle edge cases

The basic router works for most traffic, but production systems need a few refinements:

def route_request_production(
    user_message: str,
    conversation_history: list,
    force_model: str = None
) -> tuple[str, str]:
    """Production router with overrides and fallback."""

    # Allow explicit model override (for power users or testing)
    if force_model:
        model = force_model
        category = "override"
    else:
        category = classify_request(user_message)
        model = ROUTE_TABLE[category]

    try:
        response = client.chat.completions.create(
            model=model,
            messages=conversation_history + [
                {"role": "user", "content": user_message}
            ]
        )
        return response.choices[0].message.content, category

    except Exception:
        # Fallback to GLM 5.2 if the specialist is unavailable
        fallback = "glm-5.2"
        response = client.chat.completions.create(
            model=fallback,
            messages=conversation_history + [
                {"role": "user", "content": user_message}
            ]
        )
        return response.choices[0].message.content, f"{category}->fallback"

Three patterns worth noting:

Force model — Let callers bypass routing when they know what they need.
Fallback — If a specialist model is down, fall back to GLM 5.2. It handles everything reasonably well.
Return the category — Log which route each request takes. You’ll need this data to tune the system.

Cost and latency comparison

Consider a workload of 1,000 requests with this distribution: 600 simple, 300 general, 70 reasoning, 30 code. Average 500 input tokens, 200 output tokens per request.

Single-model approach (everything on V3.2)

Avg latency

~4.5s

All 1000 reqs

V3.2 only

Every request waits for V3.2’s ~1.2s TTFT plus generation time at ~60 t/s. Simple questions get the same treatment as complex analysis.

Multi-model approach (routed)

Simple (600)

~1.2s (8B)

General (300)

~4.7s (V3.2)

Reasoning (70)

~9.0s (R1)

Code (30)

~3.5s (Coder)

The weighted average latency drops to approximately 2.7s — a 40% reduction. The 600 simple requests finish in ~1.2s instead of ~4.5s. That’s a 3.7x improvement for the majority of your traffic.

The 70 reasoning requests are slower individually (~9s vs ~4.5s) because R1 generates chain-of-thought tokens. But the quality on those specific requests is significantly better — R1 scores 50.2% on HLE versus V3.2’s 39.3%.

You get faster averages and better quality on the hard tail.

Real example: a support chatbot

A customer support chatbot receives three types of requests:

FAQ (60%) — “What are your business hours?” / “How do I reset my password?”
Complex support (30%) — “I was charged twice for order #12345, can you investigate?”
Technical issues (10%) — “Your API returns 500 when I send multipart form data with UTF-8 filenames”

Without routing

All requests go to DeepSeek V3.2. FAQs get correct answers but with unnecessary latency. Technical issues get decent answers but miss edge cases that a code-specialized model would catch.

With routing

SUPPORT_ROUTES = {
    "simple":    "glm-5.2",       # FAQ, greetings
    "general":   "glm-5.2",       # Complex support
    "reasoning": "glm-5.2",       # Investigations
    "code":      "glm-5.2",       # Technical issues
    "agent":     "kimi-k2.7",     # Multi-step resolution
}

FAQs resolve quickly via GLM 5.2. Complex support issues get GLM 5.2’s full analytical capability. Technical problems also route to GLM 5.2, which understands the code context well. If a support issue requires looking up order data via API, it routes to Kimi K2.7 for tool-assisted resolution.

The classification step adds ~200ms. For the 60% of requests that drop from ~4.5s to ~1.2s, that’s an invisible cost.

When NOT to use multi-model routing

Routing adds complexity. Skip it when:

All your requests are the same type. If you’re building a code editor, just use a single coding model like GLM 5.2. No routing needed.
You have fewer than 100 requests/day. The cost savings don’t justify the engineering overhead at low volume.
Latency doesn’t matter. For batch processing or async workloads, a single capable model is simpler.
Your classification accuracy is low. If the router misclassifies frequently, you get worse results than a single good model. Test the classifier on real traffic before deploying.

The sweet spot is high-volume applications with diverse request types — chatbots, API gateways, developer tools, and customer-facing products where response time directly affects user experience.

Implementation checklist

Log your traffic. Before building a router, understand your request distribution. What percentage is simple? Complex? Code?
Start with two tiers. A fast, lighter model for simple requests, and a stronger model like MiniMax M3 for everything that needs deep reasoning, code, or long context. Add specialists only when you have data showing they help.
Measure classification accuracy. Sample 100 requests, manually label them, compare against the router’s output. Target >90% accuracy.
Add fallback. Every specialist route should fall back to GLM 5.2 if the specialist is unavailable.
Monitor per-route metrics. Track latency, cost, and quality per category. This tells you where to optimize next.

The routing pattern works with any OpenAI-compatible API. The code examples in this guide use the model ids we actually serve — GLM 5.2, MiniMax M3, and Kimi K2.7 from our Frontier pool (DeepSeek V4 Flash and MiMo v2.5 in the Core pool make great router models too); the throughput and latency comparisons cite ecosystem reference models like Llama and DeepSeek V3.2/R1 for context. If you’re building a platform that needs LLM access for your users, see how per-key plans work.

Sources: Artificial Analysis Leaderboard · DeepSeek V3.2 · HLE Leaderboard

What it takes to build your own LLM inference platform

Mar 19, 2026

If you’re building a SaaS that needs to give users access to LLMs, you have two options: build the infrastructure yourself, or use a platform that does it for you. Here’s what “build it yourself” actually looks like.

This isn’t theoretical. We built this. Here’s every component, what it does, and what alternatives exist.

0. Model access — the first problem

Before you write a single line of code, you need access to models.

Self-host on your own hardware: Buy GPUs, rent datacenter space, run the models yourself. Full control, best unit economics at scale — but massive upfront cost and you’re limited to the models you can afford to deploy. Running DeepSeek V3.2 requires multiple high-end GPUs. Running dozens of models? You’d need a data center.

Rent infrastructure: Use GPU clouds like Vast.ai, AWS, Hetzner, CoreWeave, or Lambda. No hardware to buy, but you still manage deployments, scaling, and failover. Costs add up fast — a single H100 runs $2-4/hr.

Use an inference provider: Sign agreements with DeepInfra, Together.ai, Fireworks, etc. who already have the models deployed. Pay per token, no GPU management. But you depend on their availability, pricing, and terms. If they change prices or drop a model, you need a plan B.

Mix: Most serious platforms end up here. Own hardware for high-volume models where the unit economics justify it, rented GPUs for burst capacity, and provider agreements for the long tail of models nobody runs enough to self-host.

Self-hosting dozens of models on your own is economically unrealistic. The real question is where to draw the line between own infra, rented compute, and providers.

1. Serving engine

If you self-host or rent GPUs, you need software to serve the models:

vLLM — most popular, good throughput, active community
TGI (Text Generation Inference) — Hugging Face’s solution, solid for single-model deployments
TensorRT-LLM — NVIDIA’s optimized engine, best raw performance but harder to set up
SGLang — newer, fast, good for structured generation

You’ll also need to handle model weights, quantization, scaling across GPUs, and failover when a node goes down. This is a full-time ops job.

2. API proxy layer

Your users shouldn’t hit the inference backend directly. You need a proxy that:

Translates between API formats (OpenAI, Anthropic)
Routes requests to the right model/provider
Injects authentication
Handles retries and failover
Strips provider headers so users don’t know your backend

Options:

Build from scratch with Express/Fastify + http-proxy-middleware
Use an open-source gateway: LiteLLM, Portkey, Kong AI Gateway, MLflow Gateway
Use a managed gateway: Helicone, Braintrust, Promptlayer

Each has trade-offs. Open-source gateways give you control but you manage the deployment. Managed gateways are easier but add latency and cost.

3. Authentication

Two layers:

User auth (dashboard login)

Firebase Auth, Auth0, Clerk, Supabase Auth, or roll your own
Supports email, Google, GitHub, wallet signatures

API key auth (inference requests)

Generate API keys per user
Validate on every request before proxying
Store key metadata (plan, rate limits, owner)

This is where it gets interesting for platforms. You need per-key plans — each key with its own rate limits and usage tracking. Most auth solutions don’t do this out of the box. You’ll need a custom key management layer.

4. Rate limiting

Per-key rate limiting with at least:

RPM (requests per minute)
TPM (tokens per minute)
Budget caps (dollar amount per time window)

This needs to be enforced at the proxy layer, before the request hits the inference backend. Otherwise a single user can exhaust your GPU allocation.

Options:

Redis-based counters (most common)
Token bucket algorithms
Proxy-level enforcement (some gateways include this)

If you’re using per-key plans, each key needs its own set of limits. Not one global limit — individual limits per key.

5. Usage tracking and billing

You need to know:

How many tokens each key consumed (input + output)
What model was used
Cost per request
Aggregate usage per user, per day, per billing period

For subscription billing:

Stripe for card payments
Budget windows (e.g., $X per 8-hour period)
Automatic key revocation when subscription expires

For pay-as-you-go:

Credit balance per user
Deduct per request based on token count × model price
Top-up flow (Stripe, crypto, etc.)

For crypto payments:

USDC on a supported chain
On-chain transaction verification
Wallet connector in the dashboard (wagmi, viem, etc.)

This is a significant amount of code. Usage tracking alone requires intercepting every response to count tokens, calculating cost based on the model’s pricing, and storing it per key.

6. Dashboard

Your users need a web UI to:

Create and manage API keys
View usage per key (tokens, requests, cost)
Subscribe to plans or top up credits
See available models and pricing

Tech stack typically:

React/Next.js/Vue frontend
REST API backend
Real-time usage updates

For platforms (your users creating keys for their users), you also need a management API — programmatic key creation, plan assignment, usage queries.

7. Model catalog management

Models change. New ones come out weekly. You need:

A catalog of which models you serve
Pricing per model (input/output cost per token)
Sync mechanism to update prices when providers change them
Display names, categories, tags for the dashboard
Cache pricing metadata (some models support prompt caching discounts)

This is an ongoing operational burden, not a one-time setup.

8. Documentation

Your users need:

API reference (endpoints, request/response formats)
SDK examples (Python, Node.js, at minimum)
Authentication guide
Billing/usage documentation
Quick start guide

This is easily 20-30 pages of documentation that needs to stay current.

9. Monitoring and reliability

Health checks on the inference backend
Status page for users
Alerting when latency spikes or errors increase
Logging (but not logging prompt content — privacy)
Graceful degradation when a model or provider is down

10. Compliance and privacy

Privacy policy
Data handling documentation
GDPR compliance if you serve EU users
Decision: do you store prompts? (You shouldn’t)
SOC 2 / ISO 27001 if targeting enterprise

The full stack

Component	Ongoing maintenance
Inference backend	High — scaling, failover, model updates
API proxy	Medium — format changes, new providers
Auth + key management	Low
Per-key rate limiting	Low
Usage tracking + billing	Medium — edge cases, reconciliation
Dashboard	Medium — new features, UX
Model catalog	High — weekly model updates
Documentation	Medium — keep current
Monitoring	Low
Privacy/compliance	Low

What breaks in production

Building is the easy part. The hard part is what breaks with real users:

A provider changes their API format without warning. Your proxy returns 500s for 2 hours until you notice.
A model gets deprecated. Your users’ hardcoded model IDs stop working overnight.
Token counting has an off-by-one bug. You’ve been undercharging for 3 weeks. Your margin is gone.
A user finds a way to exceed rate limits through concurrent requests. Your inference bill spikes 10x in one afternoon.
Stripe webhook fails silently. A user’s subscription expired but their API key still works. Free inference for a month.
You push a billing update and break the usage tracking. Three days of missing data. Users open tickets.

Each of these has happened to us. We fixed them. The question is whether you want to fix them yourself, with your users waiting, or use a platform that already has.

Or

You use an inference platform that already has all of this, create API keys for your users, and ship your product this week.

We built all of the above so you don’t have to. See how per-key plans work.