AI Cost Optimizer

Onboarding — start here

What this does. Turns "which model should I run this on?" into a decision with numbers attached. It reads today's OpenRouter usage rankings and today's prices, computes what your specific job costs on each candidate, and puts 2–4 real options in front of you with the saving and the risk stated. You choose; it does not choose for you.

What it does not do, and where to go instead.

• It does not tell you whether a cheap model is good enough for your task. No ranking can. It gives you a cheap way to find out — the 20-row bake-off in Step 4 — but the judgement stays with you.
• It does not optimise prompts for quality, only for size. For prompt engineering use the model docs; for Anthropic-specific behaviour use the claude-api skill.
• It does not manage keys, retries, rate limits or batching. That belongs in the job script.
• It is not a benchmark. Usage ranking says a model is paid for at scale, not that it is accurate on your data.

Setup. Needs OPENROUTER_API_KEY in the environment for pricing, and FIRECRAWL_API_KEY for the rankings scrape. Rankings are optional — without Firecrawl you still get live pricing and the job cost, just no usage signal.

set -a && . ./local.env && set +a

Verify with a one-line run. If you see a table with a rank column populated, both halves are working:

./.venv/bin/python .agents/skills/openrouter-cost-optimizer/scripts/rank_models.py \
  --in-tokens 1000 --out-tokens 200 --rows 100

How to invoke. "Which model should I use for X?", "make this cheaper", "cost this job before I run it", "I'm about to label 16,000 headlines".

Cost per run. Effectively free — one pricing API call and one Firecrawl scrape (1 credit, ~$0.001). It is always cheaper to run this than to guess wrong once.

Rest of the skill. scripts/rank_models.py does the fetching and the arithmetic. references/model-notes.md holds what the rankings cannot tell you: which labs to reach for by task shape, and the failure modes of the cheap tier.

Why this exists

A real run in this repo: $21.50 of OpenRouter spend where roughly half was avoidable. Not through anything clever — three ordinary mistakes:

Note the order. Model choice is the last lever, not the first. Halving the tokens helps on every model; switching models helps only until you switch again. Work through all three.

The method

Step 1 — Always read the live board first, and name the task

Never pick from memory, and never skip straight to price. Start every run by fetching https://openrouter.ai/rankings — the script does this for you — and identify which task the job is. The rankings page carries a "Top models by task" treemap: each task's leading models ranked by share of spend on OpenRouter.

This matters because the overall usage chart and the per-task board disagree, and the per-task board is the one you want. The headline chart is dominated by whatever is cheapest at enormous volume; the task board shows where people actually spend money for a specific kind of work. A real reading:

overall #1        deepseek/deepseek-v4-flash-0731
Classification    leading lab: openai
Data Extraction   leading lab: anthropic
Summarization     leading lab: google
Tool Dispatch     leading lab: moonshotai (Kimi)
Memory Extraction leading lab: z-ai (GLM)
Debugging         leading lab: z-ai (GLM)

Three different tasks, three different labs, none of them the overall leader. Picking the top of the headline chart for a data-extraction job is the same mistake as picking from memory, just with a fresher number.

Match the job to a task before going further:

Run without --task to print the full list from the live board.

Step 2 — Get the real token profile, do not estimate it

The single biggest source of a wrong cost estimate is guessing input size. Measure it on actual data:

sample = open("one_real_input.txt").read()
in_tokens = len(sample) / 4          # ~4 chars per token, close enough to decide

For output, take what the job actually needs to return. A classifier emitting a label and a confidence is ~30 tokens; a structured record with prose is 2,000+. If you are unsure, run one item and count.

Reasoning models bill their thinking as output. If a candidate is a reasoning model, its real output can be several times what you asked for — which is why a measured estimate can still come in under the invoice. When comparing, prefer a non-reasoning model for mechanical work; you are not paying for thought you do not need.

Step 3 — Run the ranker with the task

./.venv/bin/python .agents/skills/openrouter-cost-optimizer/scripts/rank_models.py \
  --task "Data Extraction" --in-tokens <measured> --out-tokens <measured> --rows <N>

You get every priced model costed for this job, with today's usage rank, the lab, and a * on models from the lab leading your task. It also proposes three bets, which are the shape the choice should take:

• BEST — the task-leading lab's most-used model. The capability bet.
• MID — a balance point, usually a flash/mini tier.
• CHEAPEST — the price floor that still carries a real usage signal.

Add --json to feed the numbers straight into a question.

Step 4 — Present the bets with AskUserQuestion, and never auto-pick

Cost is objective. "Good enough for this task" is a judgement about the user's data and their tolerance for error, and they own it. Spending their money on the cheapest row without asking is the same mistake as spending it on the most expensive one by reflex.

Build the options so they are genuinely different bets, not four flavours of the same one:

1. BEST — the task-leading lab. Justified when a wrong row is expensive.
2. MID — the balance point. The safe default for anything with nuance.
3. CHEAPEST — usually an open-weight or Chinese-lab model. Justified when the work is mechanical and the answer is literally present in the input.
4. The current model, priced — so the saving is a number rather than a promise. Include it even when it is obviously worst; it is the baseline the user is deciding against.
5. (when it applies) Batch endpoints — the :batch suffix is ~50% off the same model, paid for in latency (async, minutes to hours). Free money for an overnight job; useless when someone is waiting on the result.

State for each: dollar cost for this job, the multiple against the baseline, and the honest risk. Do not editorialise the cheap option into sounding reckless or the expensive one into sounding safe — put the numbers down and let them pick.

Step 5 — Bake off on 20 rows before committing to 16,000

This is the step that makes choosing a cheap model rational rather than hopeful, and it costs cents.

Run the top candidate and the incumbent over the same 20 real items. Compare on whatever the job's actual success criterion is — exact-match label, valid JSON, field-level agreement. Then decide with evidence.

agreement >= 95%   ship the cheap model
85-95%             sample 100 more, or use the cheap model with the expensive one as tiebreak
< 85%              the saving is not real; a wrong label costs more than the tokens saved

An 80%-accurate model at one-tenth the price is not a bargain when the output feeds a decision. Say that plainly if the numbers come out that way.

Step 6 — Report what was actually spent

After the job, read the real figure rather than trusting the estimate:

curl -s https://openrouter.ai/api/v1/key -H "Authorization: Bearer $OPENROUTER_API_KEY"

If the invoice beat the estimate by more than ~30%, something in the token profile was wrong — usually reasoning tokens or retries. Say so, and correct the estimate for next time rather than quietly absorbing it.

Where the cheap tier is genuinely fine, and where it is not

Ranking tells you a model is used. It cannot tell you it is right for your task. The pattern that holds up:

Cheap models do mechanical work well. Classifying a short string into a fixed set, extracting a field that is literally present, reformatting, translating, deduplicating. The answer is in the input and the job is to find it.

Cheap models degrade on judgement. Weighing contradictory evidence, deciding what a page is really selling, refusing to answer when the source is silent. Notably they are worse at leaving a field empty — a small model asked for a structured record will fill every slot rather than admit the text does not say. If your pipeline depends on honest gaps, test that specifically.

A cheap second opinion is often the right shape. Where two models must agree, the second only needs to be independent, not authoritative. Downgrading the disagreement-checker is nearly free and costs nothing in signal.

See references/model-notes.md before picking for an unusual task shape.

The three levers, in the order that pays

Run through these before accepting any quote, including this skill's own.

1. Cut the input. Scraped HTML and markdown are mostly not prose — image markdown with CDN URLs, SVG path data, nav repeated on every page, tracking query strings. Stripping it is lossless for most extraction tasks and routinely removes half the tokens. Measure before and after; do not assume.

2. Scope the output to what you read. If the code consumes nine fields, ask for nine fields. A model asked for a full record when you parse a fraction of it is billing you for text nobody will ever see. This is the most commonly missed lever because the waste is invisible in the code.

3. Then choose the model. With the job already half the size, the model decision is smaller in absolute terms — which is the point. Do it last so you are optimising the right number.

Output format

End with a short, checkable summary — the user should be able to sanity-check the arithmetic without rerunning anything:

Job:        16,000 headlines x (120 in + 60 out) = 2.9M tokens
Baseline:   anthropic/claude-haiku-4.5      $0.34
Chosen:     deepseek/deepseek-v4-flash-0731 $0.33   (rank #1 this week, 1.0x)
Bake-off:   19/20 agreement on real rows
Levers:     input already minimal; output capped at 30 tokens/row