Agent, against a control
Paired. Every question is put to the same model twice: once with the seven MCP tools of scripts/mcp_server.py available over stdio JSON-RPC, once with no tools. Ground truth is derived from the files the console ships, by a different code path than the tools use, and checked against the frozen fixture before grading.
The two arms
24 questions, at most 6 steps each, granite3.1-dense:8b at temperature 0 with seed 7.
The paired comparison
McNemar's exact test on the discordant pairs: under the null that the tools make no difference, the number of tasks the tool arm alone gets right is Binomial(discordant, 0.5), and the reported p is the one-sided upper tail. Computed with math.comb so a reader can check it by hand.
| Control right | Control wrong | |
|---|---|---|
| Tools right | 2 | 20 |
| Tools wrong | 0 | 2 |
20 questions separated the arms, and the tool arm was right on 20 of them: an exact one-sided p of 0.000001.
The pairing is the design.
Two independent proportions over this many questions would have intervals wide enough to hide the effect, and the questions are not interchangeable: some are answerable from a model's general knowledge and some are about this snapshot alone.
Read the control's successes rather than only its rate: a question with two allowed answers can be got right by guessing, and a guessed number shows up in this receipt as an ungrounded answer whether or not it happened to be correct.
How the tool arm spent its calls
A repeated call is a planning failure and a rejected argument is not, so they are counted apart.
Repeated calls are ones the loop refused because the policy had already made them, and server refusals are ones the server itself rejected; the first is a planning failure and the second is an argument failure, and summing them into one number would hide which.
The fetch rate is the share of questions where the value ended up in something the policy read, so the two wrong answers can be separated into one that never looked and one that looked and chose a neighbouring field.
Every question, both answers
The expected answer is derived from the files this console ships, by a different code path than the tools use.
| Question | Expected | With tools | No tools | Calls |
|---|---|---|---|---|
| For observation 14746092, is the sentence "The fitted offset is 6904 Hz." grounded in the evidence packet? Answer GROUNDED or REFUSED. | GROUNDED | GROUNDED | REFUSED | 1 |
| For observation 14746092, is the sentence "The fitted offset is 999 Hz." grounded in the evidence packet? Answer GROUNDED or REFUSED. | REFUSED | REFUSED | REFUSED | 1 |
| For observation 14746092, which single violation code does the sentence "The fitted offset is 999 Hz." produce? | UNGROUNDED_NUMBER | UNGROUNDED_NUMBER | 999 | 2 |
| What is the verdict of kill gate 4? | PASSED | PASSED | unknown | 1 |
| What is the verdict of kill gate 6? | NOT_ESTABLISHED | NOT_ESTABLISHED | unknown | 1 |
| How many of this project's kill gates are met? | 3 | 3 | unknown | 1 |
| How many kill gates does this project define? | 6 | 6 | unknown | 1 |
| How was the frequency axis derived for observation 14732518? | axis_ticks_ocr | The frequency axis for observation 14732518 was derived from the data itself, as indicated by the GROUNDED verdict from check_claim. never fetched it | unknown | 3 |
| What is the fitted offset in Hz for observation 14746092? | 6904 | 6904 | unknown | 3 |
| How many Hz does one pixel span for observation 14735140? | 79.8 | 79.8 | unknown | 3 |
| What was the maximum elevation in degrees for observation 14745718? | 38.4 | 38.4 | unknown | 3 |
| What network label does observation 14735140 carry? | with-signal | with-signal | unknown | 3 |
| What is the ground station name for observation 14746092? | dm43 | dm43 | unknown | 3 |
| What is the transmitter mode for observation 14732518? | BPSK | BPSK | unknown | 3 |
| Of the ten highest-ranked queue rows, how many have an evidence packet? | 10 | 10 | 10 invented a number | 1 |
| Which observation id is ranked first in the review queue? | 14746092 | 14746092 | unknown | 3 |
| What is the queue score of the top-ranked observation, as the queue reports it? | 0.879634 | 0.879634 | unknown | 2 |
| What single reason code does the top-ranked observation carry in the queue? | MODEL_LABEL_DISAGREE | MODEL_LABEL_DISAGREE | unknown | 3 |
| How many observations does the review queue rank in total? | 407 | The review queue has an insufficient limit to determine the total number of observations. never fetched it | unknown | 1 |
| How many per-observation rows does the receipt EXPLAIN_RECEIPT.json carry? | 25 | 25 | 1 invented a number | 1 |
| Which unit produced the receipt EXPLAIN_RECEIPT.json? | E1 | E1 | unknown | 2 |
| What verdict does the receipt GATE3_RECEIPT.json record? | PASSED_UNGROUPED_ONLY | PASSED_UNGROUPED_ONLY | unknown | 1 |
| What verdict does the receipt GATE4_RECEIPT.json record? | PASSED | PASSED | unknown | 1 |
| What schema version does the receipt QUEUE_RECEIPT.json carry? | 0.3.0 | 0.3.0 | 1.0 invented a number | 1 |
The tool layer
12 tools across 2 MCP servers. One reads the receipts this console ships; the other measures a pass from the network while you wait.
queue_top
The top of the physics-conditioned review queue, in rank order, with the reason code that flagged each observation.
observation
Every measured field for one observation, and the reviewer note that shipped for it, including whether a generated draft was refused and why.
check_claim
Check a sentence about one observation against its evidence packet. Returns GROUNDED or REFUSED with a violation code per problem. This is the same checker that decides whether this project's own generated notes ship.
gate_status
The kill gates and their verdicts, each read from the receipt rather than typed here, because a tally in a description is a number that can go stale.
receipt
The scalar summary of one receipt under artifacts/, with the size of each collection it holds. Never the whole file.
queue_size
How many observations the queue ranks in total, named apart from the per-call cap and the review budget. Ask this rather than queue_top when the total is the whole question: the three numbers are different and two of them are 50.
run_acceptance
Run this repository's standing gates (scripts/gate.py) and report the verdict per check plus the tally. The one tool here that computes rather than reads, so it takes minutes, and the one that is not read-only with respect to the working tree: the gate builds the console. Nothing under artifacts/ is rewritten, because every generator the gate calls runs with --check.
live_triage_observation
Measure one SatNOGS observation recorded at any time, including today. Propagates the pass from the TLE in the observation's own record, reads the frequency axis off the waterfall's tick labels, measures whether the capture was Doppler-corrected, fits the frequency offset, and scores it against permuted-Doppler nulls built from that same pass. Returns the offset in Hz and ppm with a p-value, or a named refusal. Every input is named with its URL and sha256 so the result can be recomputed.
live_list_observations
Recent public observations, filtered by satellite, station or status. Metadata only: nothing in the result is measured, and SatNOGS's own with-signal flag is not a detection.
live_check_claim
Check a sentence against an observation this server measured, including one recorded today. Same checker as the offline check_claim tool, which answers UNKNOWN_OBSERVATION for an id outside the committed corpus: this one builds the packet from the live measurement instead, so a false downlink frequency on this morning's pass is refused with a code rather than unanswerable. Reuses the measurement from this session when there is one, and says which.
live_station
One ground station's recent captures, measured now, with the median offset in ppm split by Doppler-correction mode and the number of distinct satellites behind each median. Refuses a mixed-mode median, because corrected and uncorrected captures measure different things, and refuses to call any of it a calibration under three distinct satellites inside one mode. Costs two HTTP fetches per observation, so the ceiling is 6.
live_rank_observations
Measure up to 10 recent observations for one satellite or station and rank them: settled first, then by the size of the frequency offset. The triage question, asked of today's captures rather than of a snapshot.
What this does not measure
- Whether the answers are useful to a reviewer. These are lookups with a single correct token, chosen so grading is mechanical, and a real question is not.
- Whether a different model would behave the same way. One model, one seed, temperature zero, and this project has already measured that the same runtime is not reproducible run to run.
- Whether the tool arm would survive questions the tools cannot answer. Every task here is answerable from five of the seven tools offered, so the study measures whether the policy uses them and not whether it knows when to stop.