Why do AI answers change every time I run the same question?
Last updated October 14, 2026. Written by the Third Coast AI team.
AI answers change between runs because the engines are probability machines that sample a different wording each time, decide independently on each run whether to search the web at all, rewrite your question into a different set of short search queries, and draw on a search index that is itself shifting, so SparkToro's January 2026 study of 2,961 prompt runs found less than a 1 in 100 chance that ChatGPT or Google's AI returns the same list of brands twice for the same question, which means a single screenshot of an answer is not evidence of anything and visibility has to be measured as a rate across repeated runs. The variance is large, it is documented by the engines' own descriptions of how they work, and it is the reason a vendor who shows you one answer with your competitor in it is showing you a coin toss.
This page explains where the variance comes from using the vendors' documentation, what the three largest measurement studies found about how much answers and citations move, and the method we use to measure through it, which is also the method you can use to check any vendor's claim, including ours. The measurement guide goes into the reporting side.
Where does the variance come from?
From four places, and the engines describe three of them in writing. The first is the model itself, which generates text by sampling from a distribution of likely next words, so two runs with identical inputs produce different sentences and, when the sentence is a list of recommended companies, different lists. The second is the search decision: OpenAI's web search tool guide says the model "can choose to search the web or not based on the content of the input prompt," and Google's grounding documentation for the Gemini API says the model "analyzes the prompt and determines if a Google Search can improve the answer," so the same question can be answered from training data on one run and from live pages on the next. The third is query rewriting: Google's May 2025 AI Mode announcement describes "breaking down your question into subtopics and issuing a multitude of queries simultaneously," the Gemini documentation says the model "automatically generates one or multiple search queries," and Microsoft's Copilot web search documentation says the query sent to Bing "consists of a few words informed by the user's prompt," and those rewrites vary from run to run. The fourth is the index underneath, which changes as pages are published, refreshed and demoted.
| Source of variance | Who documents it | What it does to a single answer |
|---|---|---|
| Sampling in the model | Inherent to generative models; SparkToro measured the result | Different wording and different brand lists on every run |
| Whether to search at all | OpenAI's web search tool guide; Google's grounding documentation for the Gemini API | Some runs cite live pages, some cite nothing |
| Rewriting the question into short queries | Google's May 2025 AI Mode announcement; Microsoft's Copilot web search documentation | Different pages retrieved for the same typed question |
| Movement in the search index | Sistrix and Profound drift studies | Citations earned this month may be gone next month |
| Location, account history and session | Vendor documentation on location-aware search | Two people get different answers to the same prompt |
How much do answers and citations move?
More than almost anyone assumes before measuring. SparkToro's January 2026 study, by Rand Fishkin and Patrick O'Donnell, had about 600 volunteers run 12 brand-recommendation prompts in ChatGPT, Claude and Google's AI a combined 2,961 times in late 2025, and found "a <1 in 100 chance that ChatGPT or Google's AI, if asked 100X, will give you the same list of brands in any two responses," with the same list in the same order appearing less than 1 time in 1,000. Profound's July 2025 volatility study compared identical prompts a month apart on roughly 80,000 prompts per platform and found 59.3% of Google AI Overviews' cited domains had changed, 54.1% for ChatGPT, 53.4% for Copilot and 40.5% for Perplexity, with the figures growing to 70 to 90% over six months. Sistrix's May 2026 citation drift study tracked 82,619 prompts for 17 weeks and found ChatGPT search replacing as much as 74% of its sources week to week and AI Mode about 56%, while AI Overviews replaced only about 5%, and it found that for 86% of prompts a small stable core of domains persists while the rest rotates at roughly 89% a week.
Two things in those numbers matter more than the headline. The stable core is real: SparkToro found City of Hope named in 69 of 71 ChatGPT answers about West Coast cancer hospitals, a 97% rate, and one agency named in 85 of 95 Google AI answers about e-commerce marketing consultants, so the engines do hold consistent associations between a category and a few brands even while the list around them changes. And the engines differ: a citation in AI Overviews is worth more per run than one in ChatGPT because it lasts longer, and a plan that treats the two alike will misread both.
We stopped showing clients single answers in 2025. Every result we report is a rate: this many runs, this many named you, this many cited your page, by engine, by prompt. Anything less is a screenshot, and a screenshot can be found to prove any point you like.
Third Coast AI
Why is one screenshot not evidence?
Because the probability of any one answer is low and the person taking the screenshot chose which one to keep. With a less than 1 in 100 chance of the same brand list repeating, per SparkToro, a vendor who runs a prompt ten times will see ten different lists and can pick the one where you are missing, or the one where you appear, depending on what they are selling. The same applies to your own test on a Tuesday afternoon: if you ran the question once and were absent, you have learned that you were absent on one draw, which is consistent with a true appearance rate anywhere from zero to well over half. It also applies to good news, and we have watched a client celebrate a citation that never recurred. The remedy is boring, which is why it is rarely done: run the prompt enough times to estimate a rate, in each engine, and compare rates over time.
Can you measure properly through the variance?
- Write the prompts the way buyers ask, in their words and with their constraints, and keep the wording fixed for the life of the measurement, since a changed prompt is a new measurement.
- Run each prompt with retrieval on (web search in ChatGPT, grounding in Gemini, and the live modes of AI Overviews, AI Mode, Perplexity and Copilot) and run it several times per engine in each measurement window rather than once; SparkToro's recommended read is the appearance rate across dozens of runs.
- Record four things per run: whether the engine searched, which companies it named, which pages it cited, and what it said about you, including anything wrong.
- Report the rate, not the answer: the share of runs where you were named and the share where your page was cited, by engine and by prompt, with the run count shown.
- Capture the hidden searches the engine ran behind the question where the engine exposes them (Copilot shows its queries; the API versions of ChatGPT and Gemini return them), because those queries are what your pages need to answer, and they also explain much of the variance.
- Repeat on a schedule, monthly for most companies, and read trends rather than deltas, since Profound's month-to-month drift of 40 to 60% means a single month's change is mostly noise.
- Keep a stable control set of prompts you are not working on, so you can tell your effect from the engine's drift.
That is the method behind our audit and our programs. We build a prompt bank the way buyers ask, run it with retrieval on across the engines several times per prompt, capture the hidden sub-queries and the decision factors the engines use, audit the client's pages for proof of each factor, read the pages the engines already cite, write briefs modeled on them, and re-run to verify, and we track the rates in PromptWatch and Profound alongside our own prompt miner. The tools comparison covers what the trackers do and do not capture, and the AI Overviews page explains why Google's summaries are the steadiest place to earn a citation.
Does the variance mean the work is pointless?
No, and the stable core is the proof. The brands that appear in 90% of runs for a category are not there by luck; they are the ones the engines have the most consistent evidence for, from their own pages and from what other sites say about them, and Ahrefs' 75,000-brand study found mentions on other sites correlating with AI citations three times as strongly as backlinks. The work is to move from the rotating remainder into the core, which means being the clearest answer to the hidden queries the engine writes, often enough that sampling cannot miss you, and the variance is the reason to measure carefully rather than a reason not to try. A company that expects a single answer to prove the work is done will be disappointed in both directions; a company that tracks a rate will know, within a quarter, whether the rate is moving.
Frequently asked questions
Because the model samples a different answer each run, decides each time whether to search, and rewrites your question into different short queries. SparkToro found less than a 1 in 100 chance of the same brand list repeating, so one appearance and one absence are both single draws.
Enough to estimate a rate rather than see an answer. SparkToro measured appearance rates across dozens to hundreds of runs per prompt; for a practical monthly read we run each prompt several times per engine and report the share of runs where you were named, with the count shown.
Google AI Overviews. Sistrix measured about 5% of their cited sources replaced each week, against about 56% for AI Mode and up to 74% for ChatGPT search, and Profound measured Perplexity as the least volatile month to month at 40.5%.
On its own, no. With a less than 1 in 100 repeat rate, any single answer can be found by running the prompt a few times. Ask for the run count and the rate across runs, by engine.
No, it is how the engines work. It can be measured through, with fixed prompts, repeated runs, a control set and trend reporting, and the stable core of brands that appear in most runs shows the signal is real.