Over the past months, dozens of tools have emerged promising to measure “ChatGPT rankings”, “AI positioning”, or “visibility in generative engines”. Many companies are already investing significant budgets into this new type of analytics.
But… does it make sense to measure AI visibility as if it were traditional SEO? Are ChatGPT, Google AI, or Claude consistent enough to talk about rankings?
A recent study by Rand Fishkin (SparkToro) finally provides serious data to answer these questions. And its conclusions are as uncomfortable as they are necessary.
The problem: treating LLMs like Google
For years, SEO has revolved around a relatively stable concept: ranking. One query → one SERP → a more or less consistent order.
LLMs (Large Language Models) do not work that way. They are probabilistic engines, designed to generate different responses on every run.
Still, many tools on the market are selling metrics such as:
- “You are #1 on ChatGPT”
- “You moved up 3 positions in AI”
- “AI brand rankings”
Rand Fishkin’s study shows that this approach is, at best, methodologically flawed.
The experiment: are AI recommendations consistent?
The study analyzed nearly 3,000 executions of the same prompts across:
- ChatGPT
- Claude
- Google AI Overview / AI Mode
The results are striking:
- Fewer than 1 in 100 runs returned the same list of brands.
- Fewer than 1 in 1,000 returned the same list in the same order.
- The number of results varied constantly (sometimes 3, sometimes 10 or more).
Clear conclusion:
Talking about “ranking” in AI makes no sense.
Human prompts: the great chaos
Another key finding of the study is the extreme variability of real human prompts.
People with the same intent phrase questions in radically different ways. The semantic similarity analysis shows an average similarity of just 0.081.
In other words: measuring 5 or 10 “nice-looking” prompts does not represent real-world AI usage.
So… can AI visibility be measured at all?
Here’s where it gets interesting.
Despite the apparent chaos, when many prompts are executed many times, clear patterns emerge:
- Some brands appear in 60–90% of responses.
- Others appear only occasionally.
- Some never appear at all.
This happens even with highly diverse human prompts, as long as they share the same underlying intent.
And that’s the key.
The metric that actually makes sense: percentage visibility
The study concludes that while rankings are useless, aggregated visibility does have statistical validity.
The question is not:
“What position am I in?”
But rather:
“What is the probability that my brand appears when someone asks about this?”
This completely changes the approach to SEO for LLMs:
- From positions → to probabilities
- From rankings → to presence
- From single answers → to distributions
Which AI metrics make sense (and which do not)
Metrics that do NOT make sense
- AI rankings
- Average position in ChatGPT
- Position gains or losses
These metrics ignore the probabilistic nature of LLMs.
Metrics that DO make sense
- Percentage visibility by intent
- Frequency of appearance versus competitors
- Topical coverage (which types of questions you appear in)
- Visibility trends over time
As long as they are based on many executions, not isolated snapshots.
What this means for AI visibility tracking tools
The study does not invalidate AI visibility tracking tools. But it does invalidate how some of them are marketed.
A tool is only useful if it:
- Runs prompts repeatedly
- Works with aggregated visibility, not rankings
- Groups by intent, not isolated keywords
- Shows distributions, not absolute positions
The real value is not in the dashboard, but in strategic interpretation.
Our position at Mindset Digital
At Mindset Digital, we approach SEO for LLMs with a clear stance:
- We do not sell “#1 rankings in AI”.
- We do not promise rankings that do not exist.
- We measure probability of appearance and context.
First we measure. Then we understand. Only then do we decide whether execution makes sense.
In a probabilistic environment, methodological rigor is not optional. It is the only real competitive advantage.
FAQs about SEO for LLMs and AI visibility
Yes, as long as it is understood as a visibility and probability exercise, not as a direct replacement for traditional SEO or a ranking system.
It depends on the sector and the size of the competitive space, but the study suggests that dozens or hundreds of executions are required for statistically meaningful data.
Both can be useful. Data shows that when they capture the underlying intent well, aggregated results tend to converge.
No. SEO for LLMs is complementary. AI visibility still depends largely on the same assets: content, authority, clarity, and structure.
Industries with more constrained brand spaces (healthcare, professional services, SaaS, education) tend to show more stable and actionable visibility patterns.
