What an AI Feature Costs to Run

The running cost behaves more like electricity than like software. You pay for each request. The amount depends on how much text goes in and how much comes back. The total rises with use, and there is no point at which it stops rising because you have bought enough of it.

That breaks the way most companies budget. A licence is a known number agreed once a year and forgotten. This is a variable operating cost that responds to product decisions, user behaviour and your own engineering choices, and it can move a long way without anybody shipping a new feature.

It also means the pilot tells you very little about the eventual bill. A handful of careful users asking short questions is the cheapest possible version of your feature. The expensive version arrives when everybody has it, people start pasting long documents in, and somebody builds an integration that calls it in a loop.

Four things move the cost, and it is worth knowing which one is yours. How much text you send in. How much comes back. How many separate calls each user action produces. And how often something fails and gets tried again.

Input length is where most of the surprises live, because it grows silently. A chat that resends the whole conversation on every turn gets more expensive with each message. A feature that pushes a pile of retrieved documents into the context to be on the safe side pays for all of them on every question, including the questions that needed none of them.

Call count is the one that catches engineering teams. What looks like a single question to the user can be a chain: classify the request, rewrite the search query, retrieve, generate, then check the answer against the sources. Each step is charged separately. Chains are often the right design, and they should be a decision somebody made rather than something that accumulated over a few months.

Retries deserve their own attention. A timeout that triggers an automatic retry means you pay twice for one answer. Under load, retries stack up across a whole system, and a slow afternoon at a provider turns into an unusually large invoice at the end of the month.

Costs here rise by accumulation rather than by decision. Somebody adds more documents to the retrieved set to fix one bad answer. Somebody lengthens the instructions to handle an edge case. Somebody adds a checking step after an incident. Every change was individually sensible and small, and the total is now several times what it started at.

Conversation history is the classic example. A support assistant that keeps the full context to stay helpful is paying to read everything the customer said again, on every turn, right through the conversation. Long conversations cost far more than short ones, and long conversations are usually the ones that were going badly anyway.

Then there is traffic nobody intended. Automated tests pointed at a live service. A retry loop in a client. A crawler. An internal tool somebody wired to a schedule and forgot. Put an identifier on every calling system from the first day so the bill can be broken apart, because working out afterwards which caller caused a spike is genuinely difficult.

Caching is the first lever to reach for and it helps most where the same question keeps arriving. Support is the obvious case. A small number of questions makes up a large share of the traffic, and an exact repeat can be served straight from a store without touching the model at all.

Two kinds are worth building. A plain answer cache for identical questions, which is easy and effective. And caching of the fixed portion of your input, which several providers support, where the instructions and reference material you send every time are charged at a lower rate on repeat. That only works if the stable part of your request comes first, so structure it that way from the start.

Be careful what you cache. An answer that quotes a price becomes wrong the moment the price changes, and invalidation is the part everybody underestimates. Cache aggressively where content is stable, cautiously where it moves, and never where the answer depends on who is asking.

Most systems send everything to the largest model available, because that is how the prototype was written and nobody revisited it. A large share of the work does not need it. Deciding which category a message belongs to, extracting a date, judging whether a document is relevant, rewriting a search query: a smaller and cheaper model handles all of those perfectly well.

The pattern is routing. Send the easy and common cases to something small, and the difficult or high value ones to something larger. You need a way to tell them apart, which is itself a small classification job, and you need to measure quality separately on each path so that a downgrade does not slip through unnoticed.

Do this with evidence rather than by instinct. Take your evaluation set, run it through the cheaper option, and compare properly. Sometimes the smaller model is indistinguishable on your actual work. Sometimes it fails in one specific way that matters enormously to you. There is no way to know which without running it.

Output length is the other lever, and it is the friendlier one. Asking for a short answer costs less than asking for a thorough one, and users usually prefer the short answer. Cheaper and better rarely point the same way. Take it when they do.

Evaluation is a running cost and it gets left out of nearly every estimate. Every time you change a prompt, move to a newer model, or update the documents behind the system, the test set should run again. That set costs money each time, and it needs to be big enough to be meaningful, so it is not a rounding error on the bill.

If you use a model to grade the output of another model, which is often the only practical option, you are paying for both. Grading is frequently the more expensive half, because a careful grader is given more context than the thing it is grading.

Then production monitoring. Sampling live traffic for quality checks, logging enough to investigate a complaint properly, and storing all of it somewhere. None of that is dramatic on its own. Together it is a real line item, and it is the line item that stops you shipping a regression nobody noticed.

In most of these systems the people cost more than the model does. Somebody keeps the source content correct. Somebody reviews output. Somebody handles every case the system hands over, and that queue does not shrink in proportion to the queue that disappeared.

Escalations deserve care in the sums. The cases a system passes on are the hard ones by definition, so they take longer than an average case did before you started. A feature that removes the easy half of a queue does not reduce your staffing by half, and planning as though it will is how support teams end up short handed and then blamed for it.

Include the engineering upkeep too. Providers retire model versions. Behaviour changes underneath you on an upgrade. Your own content moves. Somebody has to look after all of that, and it is a permanent fraction of a person rather than a project with an end date.

Express the cost in a unit your business already understands. Cost per support conversation. Cost per invoice processed. Cost per pull request opened. A monthly total tells you nothing about whether the feature is worth having. A cost per unit of work sits beside what that work used to cost, and the comparison makes itself.

Put hard limits in place before launch. A ceiling per user, a ceiling per day, and an alert on the rate of spending rather than only on the total, because by the time a total is exceeded the money has already gone. Every provider offers some version of this. Configure it on the first day rather than after the first surprise.

Then decide in advance what you do when the cost grows. Growth is not automatically bad. If spending rose because usage rose and the usage is valuable, the feature is working. The question worth asking every month is whether the cost per unit of work is climbing, because that means something in the system got less efficient while nobody was watching.