Skip to content

Knowledge Base

Measuring AI visibility: 63 days, 15 buyer questions

Louie Valkhof
Louie Valkhof
15 min read
Isometric 3D scene of five AI answer panels side by side, each holding a different stack of cards, with a measuring tape running past them day after day

Measuring AI visibility takes more than one question in ChatGPT: Oase Creative asked five AI systems the same buyer questions for 63 days, and on the same morning the AIs disagreed in 97 of 156 cases.

One question to ChatGPT mostly measures that morning's luck, not your AI visibility. Since 8 September 2026 we have asked AI systems the same buyer questions every morning, phrased as "which agency does X, name companies with their website". After 63 measurement days we have 802 answers to 15 buyer questions, spread across five systems: Gemini, Google AI Mode, ChatGPT, Claude and Perplexity.

The result in one paragraph. Per fixed combination of question and AI, the answer switched between "Oase Creative mentioned" and "not mentioned" in 40 of 69 cases. On the same morning and the same question, the AIs of October and November disagreed in 97 of 156 cases about whether we were named. And a single check gave a different verdict than the full series for that question in 110 of 802 answers. One measurement is not a measurement. If you want to know whether you show up in ChatGPT, you need a series, per AI.

Below is how we measured it, what shifted per question, how many measurements you need and how to set it up yourself. Every count comes from our own raw scan files. Where we do not know something, we say so.

How did we measure AI visibility?

With a fixed set of buyer questions, asked early every morning at a fixed time, without a logged-in account or memory. For each answer we store the full text, the companies named, the sources and whether Oase Creative appears in the text. "Mentioned" means our name is in the answer itself, not just in a source link underneath.

The measurement ran in two phases. From 8 September through 1 October we asked nine buyer questions to Gemini 2.5 Flash with Google Search as its search source, through the API. On 22 September that scan returned only errors; we leave that day out. From 2 October we switched to four systems at once, through the real user interfaces: Google AI Mode, ChatGPT logged out in an incognito window, Claude with web search and Perplexity on a free account without memory. The question list grew to fifteen buyer questions, which rotate: each question comes up every few days per AI, eight to eleven times in this period.

Phase System Period Usable days Buyer-question answers
1 Gemini 2.5 Flash with Google Search 8 Sep to 1 Oct 2026 23 207
2 Google AI Mode 2 Oct to 10 Nov 2026 40 160
2 ChatGPT (logged out) 3 Oct to 2 Nov 2026 31 124
2 Claude with web search 3 Oct to 10 Nov 2026 39 156
2 Perplexity (free, no memory) 3 Oct to 10 Nov 2026 39 155

One honest caveat: since 3 November the logged-out ChatGPT session in our scan has returned only empty answers. Those eight mornings do not count. Questions about us as a company ("what is Oase Creative") are also left out here; every AI answers those correctly every day, and they say nothing about buying behaviour. We keep the calculation scripts and their output next to the raw scan files, so every count in this article can be recalculated.

How often does the answer to the same question change?

More often than you would expect. In the first phase we asked the same model exactly the same nine questions for 23 days. On an average morning it named us about twice. But the daily totals ranged from one to five of the nine questions. On nine of the 23 days we appeared in just one answer, on one day in five. Had you looked once on a random morning, you could have concluded "we are almost invisible" or "we are doing reasonably well".

Per question it looks like this:

Buyer question (phase 1, Gemini) Oase mentioned Day-to-day switches Different names in 23 days Of which named only once
bol.com listing images 18 of 23 3 44 19
Amazon listing and A+ content 12 of 23 10 83 33
Product photography for a webshop 9 of 23 11 75 32
Packaging design 6 of 23 12 80 25
Brand identity and visual identity 2 of 23 4 102 58
3D product visualisations 1 of 23 2 61 24
Brand and webshop together 1 of 23 1 105 63
Brand book or brand manual 0 of 23 0 135 74
Shopify webshop with strong design 0 of 23 0 45 14

Look at the third column. On the packaging question our "mentioned yes or no" flipped twelve times across 22 day-to-day transitions. The names around us move too. Add up the different names per question and you get 730, of which 342 appeared on only one day. Of the names that appeared across two consecutive days, on average about a quarter were on both.

The sharpest example is the brand book question. On 13 September the model gave eighteen names, on 14 September nine. Two were on both lists. Over 23 days exactly one agency came back every single day.

Am I in ChatGPT? It depends on which AI and which morning

From October we asked the same questions on the same morning to four AIs. That reveals a second kind of noise: the system itself. ChatGPT named us in 82 of 124 answers, Perplexity in 59 of 155, Claude in 51 of 156 and Google AI Mode in 50 of 160. Same question, same agency, same morning, and a very different picture per screen.

Buyer question (phase 2) Google AI Mode ChatGPT Claude Perplexity
bol.com listing images 11 of 11 7 of 8 8 of 10 10 of 10
Amazon listing and A+ content 9 of 11 8 of 8 7 of 10 8 of 10
Product photography for a webshop 8 of 11 4 of 9 7 of 11 9 of 11
3D product visualisations 10 of 11 7 of 8 0 of 10 6 of 10
Brand book or brand manual 7 of 10 7 of 8 0 of 10 2 of 10
Branding and webshop for an e-commerce brand 0 of 11 8 of 9 11 of 11 7 of 11
Brand identity and visual identity 0 of 10 8 of 8 7 of 10 6 of 10
Brand and webshop together 4 of 11 9 of 9 4 of 11 1 of 11
Branding and webshop for a sports brand 0 of 10 7 of 8 7 of 10 1 of 10
Branding and website for a technical company 0 of 11 8 of 9 0 of 11 1 of 11
Brand positioning and brand strategy 0 of 10 5 of 8 0 of 10 0 of 10
Packaging design 1 of 10 0 of 8 0 of 10 7 of 10
Branding and website for a restaurant 0 of 11 2 of 8 0 of 11 1 of 11
Branding and website for a clinic 0 of 11 2 of 8 0 of 11 0 of 10
Shopify webshop with strong design 0 of 11 0 of 8 0 of 10 0 of 10

A few rows stand out. On the brand identity question ChatGPT named us eight times out of eight, Google AI Mode zero times out of ten. Packaging is the reverse: Perplexity seven out of ten, the other three almost never. On the brand book question, where Gemini never named us in September, Google AI Mode named us seven times out of ten in October and November. We do not know why it flipped. It may be the switch from Gemini through the API to Google AI Mode. Our article on what a brand book is only appeared on 16 October, while AI Mode already named us on 5 October.

Count every morning on which at least three AIs got the same question, and they disagreed in 97 of 156 cases. Asking "are we in ChatGPT?" is really asking four questions.

Why does an AI give a different list every day?

Because the answer does not come from a fixed ranking but is rebuilt every time. Google describes a "query fan-out" technique for AI Mode: for one question the system sends several sub-queries into the search engine and builds an answer from what comes back. Gemini with Google Search works the same way: the model picks its own search terms and reads what ranks at the top that day. A different search term returns different pages, and different pages return different names.

We saw this before in our piece on why ranking first on Google is not the same as being in the AI answer: in the first seventeen measurement days the model typed hundreds of its own searches, and our best-ranking keyword was not among them. On top of that, the model decides per run how many names to give. Eighteen one day, nine the next.

We are not the only ones seeing this. In January 2026 SparkToro had 600 volunteers run 12 prompts a combined 2,961 times through ChatGPT, Claude and Google's AI. The chance that ChatGPT or Google's AI, asked a hundred times, returned exactly the same list of brands twice was less than one in a hundred. Their advice matches what we see: look at how often you appear in the answer, not at your position in a single list.

Google itself says there are no additional requirements or special optimisations needed to appear in AI Overviews and AI Mode beyond regular SEO. That fits our data in one sense: the questions where we score consistently are the ones we have had good pages for over years. But it does not explain the daily swing around them. That swing sits in the system, not in your site.

How many measurements do you need for a reliable picture?

At least five per question per AI, spread over different days. That follows from a simple test on our own data. For each of the 69 series (one question, one AI) we first set the verdict over the whole series: are you mentioned more often than not? Then we repeatedly drew one, three, five or seven random answers from that series and checked whether their majority gave the same verdict.

Measurements per question per AI Matches the full-series picture
1 87%
3 93%
5 95%
7 97.5%

So a single measurement is off roughly one time in eight. That sounds acceptable, but the average hides where it goes wrong. In the 29 series that always give the same answer (always mentioned or never), one check cannot be wrong. In the 40 series that switch, 110 of the 485 individual answers gave the opposite picture. Exactly on the questions where it matters, where you are just in or just out of the frame, a single check is worth the least.

There is a second reason to keep measuring. A series also shows shifts over weeks. For us the brand book question flipped from September to October, although that coincided with our switch to other systems. With one-off checks you only see such a shift if you happen to look at the right moment, and you cannot tell it apart from a bad morning.

Five measurements is a floor for one snapshot. For a statement like "we are doing better than last month" you need two such series, with the same questions and the same setup.

How to set up measuring AI visibility yourself

You do not need a tool to start, but you do need discipline. This is the setup we use, reduced to what you can do with a spreadsheet.

  1. Write down five to ten buyer questions in your customers' words. Not "best brand agency", but the question someone actually asks: "which agency can design a visual identity for my webshop, name companies with their website". A buyer question asks for company names. An explainer question ("what is a visual identity") measures something else.
  2. Ask them under the same conditions each time. New chat, logged out or without memory, roughly the same time of day. Ask while logged in and you get an answer that leans on your own history.
  3. Measure each AI separately. At least ChatGPT and Google AI Mode; Perplexity and Claude if your customers use them. Never lump them together.
  4. Record three things per answer: are you in the text (yes/no), which other names appear, and which sources are linked. Those sources later tell you where you need to be.
  5. Repeat at least five times per question, spread over two weeks. Then calculate the share per question per AI: mentioned in so many out of so many runs.

That share is your real number. A position in the list ("we were fourth") is noise. Whether you appear, and how often, is the signal.

If you also want to know whether those mentions bring visitors, measure the other side in your own analytics. How that works in Google Analytics is explained in measuring AI referrals in GA4. Mentions and traffic are two different measurements; you need both.

What do you do with the result per question?

A series sorts your buyer questions into three buckets on its own, and each bucket asks for something different.

Always or almost always mentioned. For us that is bol.com listings and Amazon content, across all systems. Here there is little to do except keep tracking. These are the topics where we have years of work, pages and reviews. Treat it as confirmation that the basics work.

Switching. Product photography, packaging and brand identity fall here, depending on the AI. You are in the pool of candidates the model chooses from, but not firmly enough to be picked every time. This is where work pays off most: a service page that answers the buyer question literally, mentions on the lists and profiles the model reads, and clear facts an AI can take over. GEO for webshops and getting ChatGPT to recommend your webshop explain how to approach that. How small the difference between being picked and not can be is covered in how an AI recommendation tips.

Never mentioned. The Shopify question is our clearest example: zero times in 62 answers, across five systems and two months. That is no longer noise, that is a gap. Either you have no page that answers the question, or the page is hard for an AI to read, or you appear nowhere the model gets its lists from. Whether your site is accessible to AI crawlers at all, you can check against what we measured across seventy webshops in GEO SEO for webshops.

You only see the difference between bucket two and bucket three with a series. With one check, a switching question on a bad morning looks exactly like a gap.

What do we do with this measurement ourselves?

We run the scan every morning, and every month we turn it into an overview per question and per AI. Our own priorities come out of that. The Shopify question is the clearest work item, because zero out of 62 is not chance. Next come the questions for sectors such as technical companies, hospitality and clinics, where mainly ChatGPT names us now and then. We do work for clients there, but our pages do not show it enough yet.

What we no longer do is draw conclusions from one morning. On 13 September we concluded from a one-off scan that we were invisible for eight of the nine buyer questions. That was true that day. Over the full 23-day series we appeared in 49 of the 207 answers, spread over seven of the nine questions. Same data, a very different conclusion, and the second one is the useful one.

Are you a business owner who wants to know whether you show up in ChatGPT and the other AIs for the questions your customers ask? The setup above is enough to start yourself. If you want it measured properly, with the same questions every day, per AI and with the sources included, we can look at it together. It is part of our work on SEO and findability and on websites that actually answer a buyer question. Start with the free brand scan for a first picture of your brand, or tell us where you are stuck.

Louie Valkhof
Louie ValkhofFounder & Art Director, Oase Creative
Knowledge Base

Frequently asked questions

Louie Valkhof
Louie ValkhofOwner / Art DirectorAvailable for new projectsI read every message personally.
SEO

Where do you want to be found?

Name the questions and keywords your customers should find you for. We will check where you stand now, in Google and in AI answers.

Book a callWhatsApp

The first call is free.

Send a messageSubject: SEO
Start a project?