Why not use the best model on every AI task? It should perform better, right?
That is true if we look only at the model. But an AI workflow is not only the model. It also has inputs, validation, retries, budgets, permissions, tools, people reviewing the output, and real consequences when something goes wrong.
Imagine building a bridge. You need a strong design, cost estimate, prototype, construction and tests. Having the best architect available does not mean that architect should calculate every number, buy every material and inspect every bolt. Each part has a different role.
I arrived at the same conclusion while building WEE, a workflow execution engine for engineering processes.
The end-to-end workflow I validated has 21 steps. Seven are AI workers; the other fourteen are deterministic tools. The workers review code, find relevant files, propose changes and inspect the result. WEE handles repository access, files, tests, retries, branching, artifacts, budgets and the pull request flow.
I could move more of that into prompts and write less orchestration code at first. But it would make execution harder to control and explain later.
A failed test has a known set of next actions. An exhausted budget means stop. Invalid output gets rejected. Permissions are enforced by the application.
I do not need an agent to reason about those things.
That distinction changed how I think about model selection too. Ticket classification and a multi-file code change may both use an LLM, but they have different failure modes, validation requirements and costs when they fail. Sending both to the strongest available model is usually a default, not a design decision.
Price per token is not the real cost
Inference has become much cheaper.
The Stanford AI Index 2025 reports that the cost of querying a model with GPT-3.5-level performance on MMLU fell from $20 to $0.07 per million tokens between November 2022 and October 2024.
This is a capability comparison, not the price history of one specific model. Smaller models improved, hardware improved and inference became more efficient. Provider pricing also makes it clear that models from the same generation can have very different input and output costs.
Still, price per token is not the number I would optimize for.
A cheaper model can fail validation, retry, escalate to a stronger model and still end in human review. By the end, the “cheap” route may cost more and take longer. But if a smaller model already handles a bounded task reliably, a frontier model is just unnecessary capacity.
The more useful metric is cost per successful task:
effective cost =
inference
+ retries
+ validation
+ escalation
+ operational overhead
+ human intervention
---------------------------------
successfully completed tasksA study on task-specific efficiency reported 91.7% accuracy for Qwen2.5 0.5B on IMDB review classification. The 72B version scored 88.6% on the same task. The larger models did much better on mathematical reasoning in the same study.
I would not use that result to claim that smaller models are better. It supports a narrower point: if a task is constrained and already well solved, more general capability may not improve the outcome.
PrismML’s Bonsai 27B is another useful example. It is a 1-bit model at roughly 3.9 GB, and the published results show it retaining 89.5% of the FP16 model’s average score across 15 benchmarks.
Whether that remaining loss matters is a product decision. A small local model can be a good fit for offline software or workloads where data must stay local. It is probably not the right tradeoff if small accuracy losses create expensive financial reconciliation.
The question is not “is this model cheap?” It is “is it cheap enough for this task, with this validation and this failure cost?”
Make the non-AI parts deterministic first
Before choosing a smaller model, I check whether the task needs a model at all.
Schema validation, permission checks, deterministic calculations, known queries and state transitions should be software. This is one reason WEE has seven AI workers and fourteen deterministic tools. The model is there for ambiguity and judgment; everything predictable stays in code.
That also gives failures a clear meaning.
If tests fail, the workflow has a test failure. If output does not match the expected schema, it has a validation failure. If it reaches its cost limit, it stops because of policy. I would rather have these concrete events than a single generic “agent failed” status.
Once that separation exists, routing does not need to be particularly clever at the beginning. It needs to understand the work.
A purchase-order extraction can be described like this:
TaskProfile(
kind="purchase_order_extraction",
risk="medium",
capabilities={"structured_output"},
context_size="small",
validation="schema_and_reference_check",
)A repository change has different requirements:
TaskProfile(
kind="repository_change",
risk="high",
capabilities={"code", "tool_use", "long_context"},
context_size="large",
validation="tests_and_diff_review",
)These task definitions belong to the product. Model names will change. purchase_order_extraction will still mean roughly the same thing.
The policy can then define limits that make sense for that task: maximum cost, latency, attempts, and whether a person needs to approve the output. One task may allow $0.08, 12 seconds and two attempts. Another may justify $1.50, 60 seconds, three attempts and human approval.
The routing code can remain simple:
def execute(task, payload):
profile = task_classifier.profile(task, payload)
if deterministic_registry.can_handle(profile):
return deterministic_registry.execute(profile, payload)
policy = policy_store.get(profile.kind, profile.risk)
candidates = model_registry.match(
capabilities=profile.capabilities,
context_tokens=profile.context_tokens,
)
candidates = router.rank(
candidates,
profile=profile,
policy=policy,
historical_results=metrics.history(profile.kind),
)
for attempt, model in enumerate(candidates, start=1):
if attempt > policy.max_attempts:
break
if budget.would_exceed(policy.max_cost_usd):
break
result = inference.run(
model=model,
payload=payload,
timeout_ms=policy.max_latency_ms,
)
evaluation = evaluator.evaluate(
profile=profile,
payload=payload,
result=result,
)
telemetry.record(
task_kind=profile.kind,
model=model,
cost=result.cost,
latency=result.latency,
evaluation=evaluation,
)
if not evaluation.accepted:
continue
if policy.requires_human_approval:
return review_queue.submit(task, result, evaluation)
return result.output
raise ExecutionPolicyExhausted(profile.kind)I would not call this solved code. Retrying operations that can write data raises idempotency questions. Provider failures, cancellation, rate limits and concurrency need their own treatment.
But the boundary matters: product policy decides what is acceptable, the evaluator decides whether the result passed, and the inference provider runs the model. Keeping these responsibilities separate makes the model replaceable later.
Validate the output, not the model’s confidence
I would not use a model’s self-reported confidence as the main reason to accept or escalate a result. The application often has stronger evidence.
An extraction either passes its schema or it does not. Referenced IDs can be checked. SQL can be parsed and executed read-only. Code can compile, run tests and pass static analysis. A retrieved answer can be checked against the documents it cites.
These checks are not perfect, but they are about the result the product needs. For tasks without an objective check, an evaluation set with representative examples and human criteria is still more useful than trying outputs in a playground until one model feels stronger.
Research suggests that routing can work when it is tied to task quality. FrugalGPT evaluated cascades that selected models based on the request and estimated answer quality. In some experiments, it matched the best individual model with cost reductions of up to 98%.
RouteLLM, published at ICLR 2025, used preference data to route between a stronger and cheaper model. The authors reported more than a twofold cost reduction without substantial quality loss in the scenarios they tested.
Those results are useful evidence. They are not a production forecast for every system.
For example, if invoice_line_item_extraction starts looking like this:
small model
success rate: 99.4%
p95 latency: 430 ms
cost per success: $0.0018
escalation rate: 0.6%then the smaller route is probably the right default for that task.
If multi_file_bug_fix looks like this:
small model
success rate: 41%
retry rate: 38%
escalation rate: 71%
strong model
success rate: 86%
retry rate: 9%starting with the cheaper model is mostly ceremony. It may have a lower token price and still lose once retries and escalation are included.
This is where production data should replace architectural instinct.
I also want to know why a task escalated. schema_validation_failed, tool_call_failed, test_suite_failed, provider_timeout, and human_review_required tell me different things.
If purchase-order extraction moves from a 4% escalation rate to 18%, I have somewhere to investigate. The prompt may have changed. The provider may behave differently. Customers may be uploading a new format. One field may suddenly be causing most failures.
“AI spend went up” is not enough to act on.
The model should be replaceable
A hosted frontier model can make complete economic sense at low volume. Self-hosting adds GPU provisioning, upgrades, monitoring, idle capacity and engineering time. Local models may improve privacy, latency or offline support, while creating memory, hardware compatibility, distribution, update and battery constraints.
Those costs belong in the same decision.
Gartner predicts that by 2027 organizations will use small, task-specific AI models at least three times more than general-purpose LLMs. I read this as a prediction of specialization, not the end of large models.
For that to work, product logic cannot be tied to model names throughout the codebase. The product should ask for capabilities:
extract_purchase_order
classify_support_request
review_code_change
generate_customer_summaryToday, a capability may use a hosted frontier model. Later, it may use a smaller API model, a local model or deterministic software.
For each run, I still want task type, policy and prompt versions, model version, input size, cost, latency, validations, retries, escalation reason and final result. That is enough to explain most decisions when someone needs to understand why the system behaved in a certain way.
Building WEE reinforced a simple architectural preference: use software where the system can give guarantees. Use models where the task needs judgment. Spend more capability when the task and the cost of failure justify it.
The strongest model has plenty of work to do. It just should not be responsible for everything.
References
- WEE — Workflow Execution Engine.
- Stanford HAI. The 2025 AI Index Report.
- Chen, Lingjiao; Zaharia, Matei; Zou, James. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.
- Ong, Isaac et al. RouteLLM: Learning to Route LLMs from Preference Data.
- Cao, Jerry et al. Task-Specific Efficiency Analysis: When Small Language Models Outperform Larger Models.
- Gartner. Organizations Will Use Small, Task-Specific AI Models Three Times More Than General-Purpose LLMs.
- PrismML. Announcing Bonsai 27B.
- OpenAI. API pricing.