Sampling and variance in AI answers
Also known as: non-determinism, answer variance
The same question asked twice can name different companies — so AI visibility is a statistic to be sampled, never a position to be looked up.
Last revised 2026-09-07. Also available as markdown — request this URL with Accept: text/markdown.
In plain English
Ask an assistant the same question three times and you may get three different shortlists. Nothing changed on your website between the runs. The model samples as it writes, the retrieval step returns slightly different passages, and the index moves underneath both.
This is the single most important thing to understand before buying, or building, AI-visibility tracking. A tool that shows you one number for today, with no range and no sample size, is showing you a coin flip and calling it a measurement.
How to fix / set it up
- Fix your prompt set and sample it repeatedly rather than checking once.
- Report presence as a range, with the number of answers it came from.
- Judge change on a trend across cycles, never on one cycle against the last.
- Be suspicious of any tool that reports a precise AI-visibility percentage without saying how many answers it came from.
The technical detail
Treat each answer as one observation of a proportion. If you are named in 4 of 12 answers, your presence rate is 33% with a margin of error of roughly ±27 points at 95% confidence — which is to say, you have learned almost nothing. Getting that margin under ±10 points takes on the order of a hundred observations per question set.
This is why credible tracking is priced by volume and why single-run tools are free: the cost is the sampling, not the cleverness.
Report a range and a sample size, or report a trend across repeated cycles. A single figure carried to a decimal place is false precision, and the person who spots that stops trusting the whole report.
Hold everything else still while sampling: same prompts, same engines, same country, same phrasing. Variance you introduced is indistinguishable from variance you were trying to measure.
FAQ
Can I reduce the variance by setting temperature to zero?
Not through a consumer assistant, and not usefully. The retrieval step and the live index vary independently of sampling temperature, so determinism at the model layer would not give you a stable answer anyway.
How often should I sample?
Weekly is enough for most categories, because the underlying indexes move on that scale. Daily sampling mostly buys noise and cost.
Why did my score move without me changing anything?
Because that is what a non-deterministic system does. It is the reason a trend is evidence and a single reading is an anecdote.