Audit a codebase for unmetered model calls

---name: audit-unmetered-model-callsdescription: "Find model calls that bypass the gateway and therefore appear on no bill anyone reads, and detect the opposite fault of counting one call twice. Use when auditing AI spend, when a provider invoice exceeds internal numbers, before consolidating providers, or when a cost dashboard reads zero."license: MITmetadata:  title: "Audit a codebase for unmetered model calls"  version: 1.0.0  author: lobstack  category: observability  tags: [observability, audit, cost, metering]--- # Audit a codebase for unmetered model calls An unmetered call is one that reaches a provider without passing through anythingthat records what it cost. It works perfectly, it produces good output, and it isinvisible to every dashboard — so it is discovered by the invoice, weeks later,and cannot be attributed to a feature or a team. Two faults, opposite directions, both silent: - Unmetered — traffic goes direct to a provider. Internal totals are under  the invoice. Nobody can say which feature spent it.- Double-metered — one call is recorded twice by two layers. Internal totals  are over. Customers can be overcharged. This has really happened in this  codebase: every successful request was metered once by the Gateway and once by  a bridge posting the same usage to a second endpoint, inflating tokens and cost  2× in the overcharge direction. ## 1. Find the direct calls Grep for the hostnames a provider is dialled at. These are the ones the Gatewayitself uses, so they are the complete set for this deployment: ```bashrg -n --hidden -g '!node_modules' -g '!*.lock' \  -e 'api.anthropic.com' \  -e 'api.deepseek.com' \  -e 'api.groq.com' \  -e 'api.mistral.ai' \  -e 'api.moonshot.cn' \  -e 'api.openai.com' \  -e 'api.x.ai' \  -e 'dashscope-intl.aliyuncs.com' \  -e 'generativelanguage.googleapis.com'``` Then the SDK constructions, which are the same finding wearing a library: ```bash# Any client built without an explicit base URL is going direct.rg -n -g '!node_modules' \  -e 'new OpenAI\(' -e 'OpenAI\(' -e 'AsyncOpenAI\(' \  -e 'new Anthropic\(' -e 'Anthropic\(' \  -e 'ChatOpenAI\(' -e 'ChatAnthropic\(' -e 'ChatGoogleGenerativeAI\(' \  -e 'generativeai' -e 'GenerativeModel\(' # Provider credentials in the environment are the tell that survives refactors.rg -n -g '!node_modules' \  -e 'OPENAI_API_KEY' -e 'ANTHROPIC_API_KEY' -e 'GOOGLE_API_KEY' \  -e 'GEMINI_API_KEY' -e 'XAI_API_KEY' -e 'GROQ_API_KEY' \  -e 'MISTRAL_API_KEY' -e 'DEEPSEEK_API_KEY'``` For each hit, decide which it is: | finding | verdict || --- | --- || client with no `baseURL` / `base_url` | **unmetered** — route it || `baseURL` pointing at a provider host | **unmetered** — route it || `baseURL` pointing at the Gateway | metered — confirm with a live header check || a provider key still in the environment | dead credential or a path you have not found yet || a test fixture or a recorded cassette | fine; exclude it explicitly, not by accident | Places the grep will miss, so look by hand: notebooks and scratch scripts, CIjobs and cron, serverless functions in a separate repository, evaluation andbenchmark harnesses, and vendored or in-house SDK wrappers that hold the base URLone layer down. ## 2. Confirm the metered ones really meter A base URL pointing at the Gateway is necessary and not sufficient. The model keymust also be one the registry can price, or the call is served and recorded withno cost — money through the gateway, uncosted. ```bashcurl -sD /dev/stderr https://www.lobstack.ai/api/gateway/v1/chat/completions \  -H "Authorization: Bearer $LOBSTACK_API_KEY" -H 'Content-Type: application/json' \  -d '{"model":"<the key this call site sends>","messages":[{"role":"user","content":"ping"}]}' \  -o /dev/null 2>&1 | grep -iE 'x-lobstack-(priced|metered|cost-usd|model)'``` x-lobstack-priced: false is the finding. So is x-lobstack-metered: false,which means the completion happened and the ledger write did not. Server-side, the same fault shows up as summary.unpriced_requests onGET /api/v1/usage. Non-zero on a recent window means an unknown model key ispassing through right now. ## 3. Reconcile against the outside world Internal numbers agreeing with each other proves nothing. Only two comparisonsare real: 1. Ledger against the provider's own billing page, per provider, per month.   Nothing inside your system can substitute for this.2. Request count against trace count. Every request that reaches the Gateway   writes a trace row, including ones that failed before a token existed. A gap   between traces and ledger rows is a metering fault; ledger rows without traces   are a second writer. Sanity checks that have each caught a real fault here: - Is total recorded cost plausible, or is it $0.00? A table of thousands of rows  summing to zero is not a quiet month.- What fraction of rows are error rows? 99.1% of every request this Gateway had  ever recorded had failed, for six weeks, and nothing alerted — because a  failing request still returns a 200-shaped nothing to a client that does not  check.- Is the tenant id null? If every row groups into NULL, no invoice can be  attributed to anyone, and the totals are correct and useless.- Do any two rows share a request id, a session and a token count within the same  second? That is the double-metering signature. npx tsx scripts/metering-audit.ts --days 90 runs the ledger side of thisagainst a Lobstack database. ## 4. Write it down as a decision list A finding that is not a decision gets rediscovered next quarter. One line each: ```src/jobs/summarise.ts:41   direct OpenAI client, no base URL     -> route (owner: data)scripts/eval/run.py:12     direct Anthropic, eval harness        -> accept, tag as eval spendsrc/lib/legacy/chat.ts:88  Gateway, model "gpt-4-turbo"          -> unpriced key, fix the keyinfra/cron/digest.ts:7     GOOGLE_API_KEY in env, no call site   -> dead credential, revoke``` Then re-run the greps in CI so the list cannot silently regrow. The next unmeteredcall site will be added by someone who has never read this audit. ## Do not - Do not accept "our dashboard says $0" as good news.- Do not fix double-metering by deleting rows. Find the second writer.- Do not delete a provider key from the environment before you know which call  site holds it. Something will 500 in production at 3am.