How we measure AI visibility

What we measure

Every metric comes from the same raw material: the full text of an assistant’s answer to a buyer question, kept as an unedited screenshot.

Prompt sets

Questions are written the way a buyer types them, not the way a marketer would. Each set has two groups:

• Discovery questions name a category and a place, never the client. Example from the sample report: “heat pump installer Devon”.
• Branded questions ask about the client by name, to check that the facts are right.

A full audit uses a set of about 12 questions. The set is written once per client and stored with the scan. The monthly monitor re-runs the same set, word for word, so a change in the answers is not caused by a change in the questions.

Question sets are written by hand today. They are not generated or clustered automatically.

Collection

Answers are collected from the consumer web interfaces of ChatGPT, Gemini and Claude, in a browser. This is what a buyer sees, including local results and web search. It is not the same as calling the model APIs.

• A browser agent types each question into each assistant and captures the full answer.
• Each question runs in a fresh session with no chat history and no personalisation.
• Captures are taken logged out where the assistant allows it, with no sidebar or account details in frame.
• Sessions run from the client’s market. The US scans ran from US servers.
• An audit is one run. The monitor repeats the run monthly.

The research pipeline collects differently. Studies and the demo dashboard call the vendor APIs with web search switched on and the user location set to the market. Every question runs twice per model. Each answer is stored with its full API response in an append-only SQLite database, so any published number can be traced to the raw answer. Cited URLs are reduced to domains and classified by a readable rule list, and a stability score reports how much the cited sources overlap between the two repeats. The first study is who AI assistants cite for health questions.

For client audits, run-to-run variance is not yet averaged out. Each question is asked once per assistant per run. See Limits.

Extraction

The agent reads each answer and records which companies were named, in what order, and what was said about the client. The result is one structured file per scan, scan.json, with a fixed schema: score, grade, per-assistant findings, named competitors and a fix list.

Accuracy is checked by a person. Every recorded company name and quote is compared with the screenshot it came from before the report is built. There is no automated accuracy score yet.

Reporting

A build script turns scan.json and its screenshots into a static report page and a PDF. The script only binds data to a fixed template. It does not write findings.

• Automated: schema validation, image checks, page build, PDF render, publishing through a GitHub Action.
• Agent-drafted, human-reviewed: the findings, the competitor notes and the fix list.
• Human only: the decision to publish, and every email to a client.

The monthly monitor uses the same process and the same question set. It has no separate automation in the codebase yet.

Controls

• A person approves every report before it is published or sent.
• The build fails if a required field is missing or a screenshot file is corrupt. A broken report cannot be published by accident.
• Agents upload screenshots through a write-only store. They cannot read, list or delete files, and they have no server access.
• Client report pages are not indexed and use slugs that cannot be guessed.
• Companies named in an answer are not contacted and are not told they were named.
• Agents do not send email, do not log in to client accounts and do not spend money.

Every change to a scan or a report is a git commit, so there is a dated record of what was published and when.

Architecture