metrics-mcp — a semantic layer an agent has to go through
Metrics defined once in YAML, SQL built from dbt, and every number an agent states checked against the result it came from before the sentence is allowed out.
- 11
- metrics, 43 column refs checked
- 100%
- invented numbers caught
- 90%
- wrong-month claims caught
- 0%
- caught without month scoping
Why
Point an agent at a warehouse with a SQL tool and it will answer every question you ask it. Some of those answers will be wrong in a way nobody catches, because the two failures that matter do not look like failures.
It writes its own definition. Asked for active accounts twice in one conversation, it can produce two different numbers — one counting any event, one counting meaningful ones — and both look right.
It states a number that is not in the data. A figure near the real one is indistinguishable from the real one at a glance, and it only has to happen once for the whole surface to stop being trusted.
This closes both structurally, rather than by asking nicely in a prompt.
The design
A metric is defined once, in YAML: definition, grain, model, columns, caveats.
The agent does not write SQL — it names a registry entry, and the statement is
built from that entry against a dbt model. There is no run_sql tool, and a
test asserts the exposed tool set is exactly four, so adding a fifth is a
deliberate act with a red build attached.
A ratio is stored as a pair of columns, never as a rate. sum(numerator) / sum(denominator) survives aggregation; avg(rate) does not, and it is the one
people reach for — it gives an account with one user the same weight as one
with forty, and the number stops being what its name says while still looking
fine. The registry cannot express a ratio any other way, and a test asserts the
two roll-ups disagree, so the reason for the rule is demonstrated rather than
asserted.
The registry is checked against the warehouse, not trusted: every model and column reference is resolved on each build, so a renamed column fails CI instead of quietly returning a wrong number to whoever asks next.
Watching the guard fail
Grounding at 100% means nothing on its own — a check that always passes scores 100% too. So two stubs answer wrongly on purpose, in the two different ways an answer can be wrong.
| how the stub lies | membership check | scoped to the named month |
|---|---|---|
| a number that is nowhere in the data | 100% caught | 100% caught |
| a real number, from the wrong month | 0% caught | 90% caught |
That second row is the finding. An invented number is easy — it is absent from the result and any check finds it. A real number attached to the wrong month passes a membership check every single time, and it does not look invented to a human reader either. Both figures are gated in CI.
Building this also surfaced a bug in the checker itself, one that only the deliberately lying stub could find. It has its own write-up: a check that always passes scores 100%.
Why it exists
It is the pattern I run in production at work: a registry of metrics in git, a dbt project under it, and an LLM commentary layer whose numbers are verified before they ship. That code is not mine to publish, so this is the same idea rebuilt small on a generated dataset, where every part can be read.
- Python
- MCP
- dbt
- DuckDB
- Pydantic
- pytest
- GitHub Actions