AI Search Visibility Using SEO Datasets
Crawlers took our new SEO datasets within hours, and GPTBot now reads the catalog on four of every five sites it visits. The jump after September 23 was a burst, not a new level. Still no Google clicks on a dataset page, and no log can show a citation.
GPTBot takes the catalog on 97 of the 120 sites it reads, and the September 23 jump was a burst, not a new level.
AI Search Visibility: results, limits and next checks
Retrieval confirmed ... burst measured ... citation openBefore the AI Search Visibility work
Before September 2026, AI Symantix published its measurements as prose. The portfolio declared datasets only inside each site's catalog file, and no one had checked whether crawlers took the data files a page linked, or whether declaring datasets on every page changed anything.
What changed in AI Search Visibility
AI Symantix published five SEO datasets between September 7 and 12, each as a page, a JSON file and Dataset structured data naming the file. On September 23 every rendered page on the portfolio began carrying DataCatalog and Dataset JSON-LD pointing at its catalog, under Digital Karma Federation v8.2. On October 11 AI Symantix added a sixth data file with this case study, and on October 20 it published four more.
What the AI Search Visibility results show
Every AI Symantix data file was fetched within hours of publishing, and the referers show Googlebot and OAI-SearchBot following the page link to the file. After September 23, GPTBot's catalog reach went from 46 sites in 11 days to 84, OAI-SearchBot's from 10 to 33 and Meta's from 1 to 18. In the windows since, GPTBot fell back to 53 and then 24 in seven days while OAI-SearchBot held at 11 to 13. GPTBot takes the catalog on 97 of the 120 sites it reads. With the BellyUp! sites removed, OAI-SearchBot spends 3.4 percent of its successful requests on the dataset layer and GPTBot 3.1 percent.
What Google reported for AI Search Visibility
Google showed the seven AI Symantix dataset pages 396 times between August 10 and October 19, for zero clicks. The CTR dataset page has 141 impressions at position 49.9, and its top query is still the pasted header row of a Search Console export. Weekly impressions for the whole site went from 236 to 3,687 and the searches showing it from 25 to 300, with weighted position at 35.4 and about three clicks a week.
What the AI Search Visibility server logs recorded
Thirty-five retained days, September 17 to October 21, 2026, on the 127 sites outside the BellyUp! family. Googlebot fetched /data/lab/exposure-velocity.json 22 times, every time with /ai-symantix-lab/ as referer. OAI-SearchBot took all six dataset files in three seconds on October 11 with /datasets/ as referer. Bots that could not be named made 14,727 of the 27,318 dataset-layer requests. Catalog history by crawler runs back to June 11.
What the AI Search Visibility results do not prove
A successful request proves a request reached the server, not indexing, citation, training use or a reader. The window comparisons are sequences, not proof of cause: the September 23 change shipped alongside other work, and crawler schedules cannot be seen. Server logs older than 35 days are gone, so the September 12 GPTBot referer record stays as first published. Human counts may include our own visits. The BellyUp! sites are excluded from every count because their finder pages create crawl loops that swamp any ratio.
Where AI Search Visibility stands now
Retrieval of published SEO datasets is confirmed across search engines and AI crawlers. The September 23 jump in catalog fetches was a burst. GPTBot and OAI-SearchBot now take the catalog on most of the sites they read, and the two OpenAI crawlers spend about the same share of their requests on dataset files. No citation has been observed, and no finding here claims one.
What we will measure next for AI Search Visibility
Re-count catalog reach per crawler on November 10 to see whether OAI-SearchBot stays above its starting level. Read the November 1 Search Visibility Index for AI Symantix. Run a fixed-question citation test for the datasets in at least two AI tools, which has not started.
What we measured for AI Search Visibility
Three readings from the October 22 re-count. Each one is a retrieval fact, and none of them is a citation.
The crawler did not find the file by guessing. It read the page, then took the file the page linked. On October 11 OAI-SearchBot opened the datasets page and took all six files within three seconds.
OAI-SearchBot does it on 56 of 84 sites, Googlebot on 71 of 124 and ClaudeBot on 12 of 117. Amazonbot has not taken the catalog on any of the 126 sites it reads.
OAI-SearchBot went from 10 to 33 and Meta from 1 to 18. Then GPTBot settled back to 53 and 24 in the last seven days, while OAI-SearchBot held at 11 to 13. A burst with a date, not a new level.
The dataset layer was 0.8 percent of the crawl on the 127 sites outside the BellyUp! family: 27,318 requests against 3.45 million content page requests. Small, steady and measurable is still the honest description.
AI Search Visibility: the short version
AI Symantix published five SEO datasets in September 2026, and on September 23 the whole portfolio began declaring datasets on every page. This case study joins the warehouse server logs with Search Console data to show who fetched the files, how they found them, what the September 23 change did, and what Google did with the dataset pages.
First published October 11. Re-counted October 22 with a newer log window, catalog history back to June 11 and a fix to one comparison the first version got wrong.
AI Search Visibility in plain English
We put our own SEO numbers online as public data files. Crawlers took the new files within hours, and the logs show them following the link on the page to get there.
When every page began labeling its datasets on September 23, GPTBot made a big grab for the catalog. Within three weeks it was back near its old pace.
We also fixed a mistake from the first version. One comparison between crawlers was thrown off by a group of sites that crawlers loop through for days.
None of this has turned into Google clicks, and the logs cannot tell us whether any AI quoted us.
Sites Where OpenAI Crawlers Fetched the Catalog
Distinct portfolio sites with a successful /ai/catalog.json request in each window, BellyUp! sites excluded. Windows are 11 days long except the last, which is the seven days to October 21, 2026.
GPTBot went 46, then 84 right after September 23, then 53, then 24 in seven days. OAI-SearchBot went 10, then 33, then 13 and 11. The burst came with the change, and OAI-SearchBot stayed above where it started.
Sites with a catalog fetch
Share of Sites Where a Crawler Also Took the Catalog
For each crawler, the sites where it fetched /ai/catalog.json divided by the sites where it read pages. September 17 to October 21, 2026, BellyUp! sites excluded.
The two OpenAI crawlers lead, with Googlebot next. Amazonbot read pages on 126 sites and took the catalog on none.
Percent of the sites a crawler reads
AI Symantix Google Impressions by Week
Search Console query rows, weeks start Monday, position weighted by impressions. The first week has six days. Searches means different queries that showed the site.
Weekly impressions grew 16 times and the site now shows up for 300 different searches a week, while average position held in the 30s.
Google impressions per week
AI Search Visibility: the starting conditions
In September 2026 AI Symantix published five SEO datasets in a row: a click-through-rate curve by Google position, a monthly Search Visibility Index, two AI bot visibility studies and a search-to-inquiry baseline. Each one shipped three ways at once, as a readable page, a downloadable JSON file, and Dataset structured data naming that file. All five were built from the Digital Karma Data Warehouse, which imports Search Console data and classified server logs for every site in the portfolio every night.
Then on September 23 the whole portfolio joined in. Digital Karma Federation v8.2, the standard kept at DigitalKarmaWeb.com, made every rendered page on every site carry a DataCatalog node and Dataset nodes pointing at that site's catalog file. That gave the warehouse a clean before and after.
We first wrote this up on October 11. The question was the one everyone asks after publishing data for AI: did anything come and get it, and did it matter?
On October 22 we ran the count again. The warehouse keeps 35 days of server logs, so the window moved forward to September 17 through October 21. The crawler history it keeps for catalog fetches reaches back to June 11, which let us check the September 23 jump against three months of earlier behavior instead of two weeks.
The site-level write-up with every table is AI Search Visibility Using SEO Datasets on AI Symantix. This entry is the portfolio receipt.
What changed in AI Search Visibility
Server logs cover September 17 through October 21, 2026, the 35 days the warehouse retains. The first version covered September 6 through October 10, and those older days have since aged out. Catalog fetch history by crawler comes from the warehouse's page-level crawler rollup, which runs back to June 11. It reproduced the first version's catalog site counts exactly before we used it for anything new. Search Console data for AI Symantix runs from June 16 through October 19, 2026.
The portfolio in this version means the 127 websites outside the BellyUp! family. We left the family's 121 sites out of every count, on both sides of every ratio. The findings explain why.
The dataset layer means successful GET requests to five kinds of file:
- The catalog at /ai/catalog.json
- The manifest at /ai/manifest.json
- The llms.txt and llm.txt guidance files
- The federation files and any other JSON under /ai/
- Public JSON data files under /data/, /content/ and /datasets/
Crawlers were named from the warehouse fingerprint table and grouped by the job their owner publishes for them: search, AI search, AI training, or fetching on behalf of a person. Requests classified as a bot with no matching fingerprint are reported separately as bots that could not be named.
A crawler's share is its dataset-layer requests divided by those requests plus its successful page requests. Catalog reach counts the distinct sites where a crawler requested the catalog in a window. Windows are 11 days long so they compare fairly. Weighted position is position weighted by impressions, and weekly rows start on Monday.
What the AI Search Visibility results show
GPTBot takes the catalog on 97 of the 120 sites it reads, and the September 23 jump was a burst, not a new level.
Crawlers Took the New SEO Dataset Files Within Hours
AI Symantix published four more research files on October 20 at 13:01 UTC. ClaudeBot fetched all four, starting 2 hours and 44 minutes later and finishing 12 minutes after that. GoogleOther came between 3 and 4 hours after publishing. DataForSeoBot followed the next day.
The October 11 file went the same way. ClaudeBot had it 3 hours and 16 minutes after it went live, and GoogleOther about 4 hours after. That night at 20:36 UTC, OAI-SearchBot opened our datasets page and took all six dataset files within three seconds. Every request carried the datasets page as its referer.
Googlebot is the steady one. It fetched the Exposure Velocity file 22 times between September 19 and October 19, and every fetch came from the Lab page. The first version counted 18 in an earlier 35 days. Same page, same file, same habit.
The first version also recorded GPTBot following a link from Signal Architect Group to our baseline file on September 12. Those log days have aged out of the warehouse, so that record stays as first written and cannot be re-run.
The September 23 Catalog Jump Was a Burst
On September 24 GPTBot fetched the catalog on 28 sites. On September 25 it did it again on 28 sites. Its best day in the three months before that was 16 sites, in July.
Counted in equal 11-day windows, GPTBot reached the catalog on 46 sites just before September 23 and 84 just after. The window after that it was back at 53, and the last seven days reached 24. OAI-SearchBot went from 10 sites to 33, then 13, then 11 in seven days. Meta's crawler did the same thing smaller: 1 site, then 18, then 3.
The first version called this more than doubled and a strong hint. With history back to June, it reads as a burst. The crawlers came to look at the new labels, and then went back to their own schedules.
One crawler held above where it started. OAI-SearchBot has reached the catalog on 11 to 13 sites per window since the burst, against 4 to 10 before it. GPTBot also made a second sweep on October 11, when it took the catalog on 24 sites in a single day.
Over equal 29-day windows, GPTBot's reach went from 58 sites to 91, and OAI-SearchBot's from 16 to 52. Most of that gain sits inside the burst.
Which Crawlers Take the SEO Dataset Catalog
The cleanest count asks one question per crawler. On the sites where it read pages, on how many did it also take the catalog? Crawl loops cannot inflate that number, because a site counts once.
GPTBot takes the catalog on 97 of the 120 sites it reads. Amazonbot read pages on 126 sites and has not taken the catalog once. The second chart shows every crawler.
What We Got Wrong About Search Crawlers and Training Crawlers
The October 11 version said OAI-SearchBot spends one visit in 25 on the dataset layer and GPTBot one visit in 2,300. It built a conclusion on that: crawlers that answer questions want the catalog, and crawlers that collect text want pages.
That comparison was wrong.
GPTBot made 11,968,879 requests across the portfolio from September 17 to October 21. All but 35,313 of them landed on BellyUp! finder pages, where it loops through filter buttons. That is 99.7 percent of its crawl, and one 28-page site gave it 6.84 gigabytes in twelve days. The first version left the BellyUp! sites out of the before and after table and left them in the share table. GPTBot's page count was almost entirely loops.
The BellyUp! sites served 24.1 million of the portfolio's 27.5 million content requests in those 35 days. That is 87.5 percent of everything.
With the BellyUp! sites out of both sides, the two OpenAI crawlers are level. OAI-SearchBot spends 3.4 percent of its successful requests on the dataset layer. GPTBot spends 3.1 percent. Applebot spends 2.8 and Bingbot 1.7. PetalBot and Meta's crawler sit near 1. Googlebot and ClaudeBot sit near 0.4. Amazonbot and PerplexityBot sit at 0.2.
So the old line does not hold. The split is not search crawlers against training crawlers. It follows the company more than the job. The first version drew a clean line, and a denominator drew it, not the crawlers. The re-count caught it because the catalog history was there to check against.
Who Fetches the SEO Dataset Layer
The dataset layer drew 27,318 successful requests in 35 days from the 127 sites outside the BellyUp! family. The same sites served 3.45 million content page requests. That is 0.8 percent of the crawl. The first version said three tenths of one percent because the BellyUp! sites had swollen the page count.
Eight groups fetched it:
- Bots the warehouse could not name: 14,727 requests
- People: 4,149 requests
- SEO tools, monitors and other named bots: 2,974 requests
- Scanners: 2,383 requests
- Search engine crawlers: 1,699 requests
- AI training crawlers: 1,146 requests
- AI search crawlers: 225 requests
- AI tools fetching for a person: 15 requests
People read llms.txt and llm.txt 1,797 times from 1,274 different addresses. We wrote those files for AI. People are reading them too. Human counts can include our own visits.
Google Still Sends No Clicks to an SEO Dataset Page
Google showed the seven AI Symantix dataset pages 396 times between August 10 and October 19, up from 233 through October 8. Clicks to those pages: zero.
The CTR by Search Position page grew from 57 impressions to 141. The search "ctr by position" now shows it at position 54, up from 66. The strangest query is still there too. The header row of a Search Console export, pasted into Google with the commas in it, has now shown the page 64 times.
Over on DatasetsMaker, the SEO Keyword and Ranking Dataset page has 815 impressions and one click since July 28. The search "seo datasets" shows it at position 62 on 172 impressions. The search "seo dataset" shows it at position 74 on 131. Those positions have not moved since October 11. People search for SEO datasets. Our pages sit at the depths where nobody clicks.
How AI Symantix Search Visibility Moved Around the Datasets
The datasets were one part of a bigger push on this site. The checker, the glossary cluster and the page consolidation landed in the same weeks, so the datasets cannot take credit for what moved. The third chart shows what moved.
Weekly impressions went from 236 to 3,687. The number of different searches that showed the site went from 25 to 300. Weighted position went from 88.2 to 30.3 in the week of October 5, then settled at 35.4 as the site entered more searches.
The AI search visibility searches have the same shape. In the week of June 15 they averaged position 89.9 across 5 searches. In the week of October 5 they averaged 27.0 across 17. The week after that the site entered 28 of them on 421 impressions and the average settled at 40.4. More searches at a deeper average is the usual shape of a foothold before the climb.
Clicks are still about three a week. Of the 21 clicks since June 16, 17 came from people typing the brand name. The CTR by Search Position dataset puts positions 21 to 50 at 0.15 percent. At 3,687 impressions that predicts five or six clicks. The site got three. The dataset we published to explain click-through rate ended up explaining our own.
AI crawler visits to AI Symantix itself rose from 362 in June to 1,754 in September. October is running at 69 a day against 58 in September.
What SEO Dataset Fetches Cannot Show
AI search visibility has three parts: retrieved, cited and accurately represented. Everything here lives in the first part. A fetch proves a crawler asked for a file and got it. It does not prove the file was kept, used in an answer or described correctly.
The closest thing to a citation signal is an AI tool fetching a file while it answers a person. ChatGPT-User and Claude-User did that 15 times across five sites in the 35 days. That is not nothing, and it is not a citation.
The citation test is still open. The first free AI answer tool we tried asked for an account before it would answer, so no test has run yet. Nothing in this case study claims a citation.
What AI Search Visibility teaches us
Publishing SEO datasets gets them retrieved, and fast. Named crawlers fetched each new file within hours, and Googlebot keeps coming back to the same file because the page links it. Put the catalog on every page and the crawlers answer with a burst of catalog fetches. Then they settle into their own rhythm. GPTBot ends up reading the catalog on four of every five sites it visits.
What the logs do not show is clicks or citations. Google lists a dataset page like any other page, at the position the page earns, and the Search Console Dataset report only counts pages that carry the Dataset label. A fetch is the first of three parts. The other two still have to be tested by asking AI tools the question and reading the answer.
The method held up because it could be checked. Count sites, not requests, wherever crawlers loop. Keep the catalog history, because raw logs only last 35 days. And run the count twice: the second reading fixed a headline the first one got wrong.
AI Search Visibility: the full analysis
What Changed Since October 11
- The crawler comparison was rebuilt without the BellyUp! sites. OAI-SearchBot at 3.9 percent against GPTBot at 0.04 percent is now 3.4 percent against 3.1 percent.
- The catalog jump is now described as a burst. GPTBot's reach was 46, 84 and then 53 sites in three equal 11-day windows.
- The log window moved from September 6 through October 10 to September 17 through October 21. The portfolio count moved from 190 sites to the 127 outside the BellyUp! family.
- Four research files published on October 20 and the first fetch times for named crawlers were added.
- Search Console data now runs through October 19. Dataset page impressions rose from 233 to 396. Clicks stayed at zero.
- Googlebot's count for the Exposure Velocity file is 22 in the new window, up from 18 in the old one.
Explore More Cases Like AI Search Visibility
Compare the starting conditions, changes and results across the case study library.