Routing two LLMs without going broke
No meta-router, no SDK, no tenth provider. Two bare fetch() calls, one timeout, one fallback, and muffin quotas that turn every AI call into a spending cap. A guided tour of Grimoire Culinaire's LLM routing.
The meta-router trap
When I started on the AI features of Grimoire Culinaire, my recipe app, the temptation was the classic one: build a nice multi-provider abstraction on day 1. A generic interface, N adapters, a smart router, maybe LangChain to look serious. Everyone does it, so it must be the right answer.
I did the opposite. Grimoire's AI client fits in a single file inside a Deno Edge Function: two functions, one bare fetch() to OpenAI, one bare fetch() to Gemini. Zero SDK, zero orchestration framework. The doctrine is written as a comment at the top of the file: API keys never in the app, Structured Outputs so there is no risky parsing, automatic OpenAI-to-Gemini fallback, minimal logging of every call.
An LLM provider is an HTTP API. Two providers means two HTTP calls. The meta-router I did not build would have solved a problem I do not have: I do not have ten providers, I have two, chosen for precise reasons.
Primary, fallback, timeout: that is all
The primary is OpenAI gpt-4o-mini, called with Structured Outputs: I pass a strict JSON schema and the API guarantees the response respects it. For structuring a recipe dictated by voice or pasted from a website, that is exactly what I need: I parse the result with no safety net and no rescue regex.
The fallback is Gemini 2.0 Flash. And it is structurally less reliable, which I accept: Gemini does not receive the schema as an API constraint, it receives it injected into the prompt, with a JSON responseMimeType and a prayer. It works almost every time. Almost. That is the price of a fallback: it is there to save the request when the primary goes down, not to be as good as the primary.
Between the two, a single switching mechanism: a Promise.race between the OpenAI call and a timer.
// supabase/functions/_shared/ai-client.ts (simplified excerpt)
/** Timeout for the OpenAI call.
* 30s because Structured Outputs with a full schema + long text
* regularly take 5-15s, and the Docker network adds latency. */
const OPENAI_TIMEOUT_MS = 30_000;
export async function callAiWithFallback(systemPrompt, userContent, jsonSchema) {
try {
return await Promise.race([
callOpenAI(systemPrompt, userContent, jsonSchema),
new Promise((_, reject) =>
setTimeout(() => reject(new Error("OpenAI timeout")), OPENAI_TIMEOUT_MS),
),
]);
} catch (openaiError) {
// Primary failed or too slow: try the backup.
}
return await callGemini(systemPrompt, userContent, jsonSchema);
}The routing flips depending on the task
Here is the detail that kills the idea of a "best provider": it does not exist. Only the best provider per task and per price exists.
For everything text-based, OpenAI is primary and Gemini is backup. But for photo OCR, when the user photographs a handwritten recipe from their grandmother, the order flips: Gemini Vision becomes the primary and GPT-4o Vision the fallback. On that specific task, at my volume, Gemini reads better for less money. The ai-vision file states it in its header comment, and the code does exactly that: Gemini first, OpenAI if it breaks.
So two providers does not mean a frozen hierarchy. It means two building blocks, and a routing decision per call type. That is the whole point of keeping the client simple: flipping a preference order means flipping two lines, not reconfiguring a framework.
Muffins as a spending cap
The real protection against going broke is not in the AI client. It lives in the app's internal currency: muffins.
Every AI action has a fixed cost, declared in an enum on the app side and enforced server-side: structuring a recipe costs 15 muffins, estimating nutrition 5, a photo OCR 20, a meal plan 70. The user has a wallet, debited before the call. The Edge Function's doctrine is explicit: quota check before the AI call, and never debit if the user did not receive a usable result.
The consequence is mechanical. A user cannot trigger more calls than their wallet allows, and every call has a bounded token cost. My financial exposure per user is therefore capped by construction, not by monitoring.
// app/lib/shared/constants/muffin_pricing.dart (simplified excerpt)
enum AiActionCost {
structureRecipe(15), // structure a recipe (text or voice)
nutritionEstimate(5), // nutrition estimate
ocrRecipe(20), // photo OCR via Gemini Vision
mealPlan(70); // 7-day meal plan
const AiActionCost(this.cost);
final int cost;
}Steering without redeploying, and the confession that comes with it
The last layer is a configuration table in the database, filled at seed time: primary provider, model, backup provider, timeout, max input length. The idea: being able to swap models or tighten a parameter from the admin panel, without redeploying an Edge Function.
And this is where I have to confess something. While rereading the code for this article, I found an inconsistency: the seed declares ai.timeout_ms as 10000, ten seconds. The code, meanwhile, hardcodes OPENAI_TIMEOUT_MS at 30,000, with a comment that justifies the 30 seconds very well. The two values have never met: the AI client does not read that config entry. The "steerable from the admin" layer exists in the database, but for the timeout, the wiring was never done.
I could quietly fix it before publishing. I would rather leave it as an example: a config the code does not read is worse than a hardcoded value, because it gives the illusion of control. Whoever changes 10000 to 5000 in the admin panel will believe they tightened the timeout. They will have changed nothing at all.
-- supabase/seed.sql (excerpt)
('ai.primary_provider', '"openai"', 'Primary AI provider'),
('ai.primary_model', '"gpt-4o-mini"', 'Primary AI model'),
('ai.fallback_provider','"gemini"', 'Backup AI provider'),
('ai.timeout_ms', '10000', 'Timeout before fallback (ms)'),
-- ... while the code says: const OPENAI_TIMEOUT_MS = 30_000;What I take away
The reliability of Grimoire's AI layer does not come from the number of providers or from a clever abstraction. It comes from the sharpness of the contract: a JSON schema enforced on the primary, an explicit timeout, a fallback accepted as less good, a fixed cost debited before every call.
Two providers are enough when each has a clear role, and the role can flip depending on the task: OpenAI up front for text, Gemini up front for vision. The most robust spending cap is not a billing alert, it is a wallet debited before the call.
And the bonus lesson, courtesy of my own repo: every "steerable" parameter must prove it is actually read by the code. My timeout, configurable at 10 seconds, has in fact been running at 30 from the start. Nobody had noticed, and that is exactly the problem.