This Week at Real SEO Life: Page One at Last, and We Caught Ourselves Causing the Robot Problem

On October 11 I said the giant rise in robot visits was not the internet suddenly loving us. It was four robots walking one enormous pile of web addresses. Then I listed what would prove me wrong.

Two checks later, on October 17 and October 24:

➤ I said four robots would keep more than 90 percent of all robot visits. They held 97.96 percent, then 99.39 percent. ➤ I said our seven main sites would stay between 1,002 and 2,004 doubled, so 4,008 visits a day. They came in at 2,010, then 1,972, then 1,403. ➤ I said OpenAI's search robot would stay near 32,110 visits. It is at 35,802. ➤ I said Perplexity's robot would stay near 7,469. It is at 10,332. ➤ I said ChatGPT's fetch robot would stay near 9,340. It is at 9,139.

Five for five, twice.

Why this matters to you: anybody can explain a chart after the fact. Writing down beforehand what would prove you wrong is the part that makes it worth anything. It costs nothing. Try it on your next hunch.

AISymantix.com is our site about measuring AI visibility. Two weeks ago I told you one of its searches had climbed from 73rd to 12th and I was pleased about it. It kept going:

➤ "searchmetrics seo visibility": now 6th ➤ "seo visibility searchmetrics": now 9th ➤ "searchmetrics visibility": 39th to 12th ➤ "searchmetrics website analyzer": brand new, straight in at 14th ➤ "seo visibility score": brand new at 24th

Sixth place is page one. That is the first time any page in this group has been there for a search people actually type.

Eight searches appeared for that site that were not there a month ago. Six more improved, one got worse, none dropped off.

My favourite thing this week. While all that was happening, AISymantix's average spot in Google went from about 31st to about 37th. On paper, backwards.

Two weeks ago I explained why that number fools people, and I used it in the other direction. Now I get to use it in this one.

Average position is your average across every search you showed up in. Eight new searches arrived. Some of them put us 55th or 61st. Adding a pile of new searches where you rank badly drags your average down, even while you are climbing on everything you already had.

Same class as last time. Last week some low scoring kids transferred out and the average went up. This week a bunch of new kids enrolled and the average went down. Nobody got dumber either time.

The real signal is in the other column. Impressions, which is how many times we appeared at all, went from 139 a day to 527 a day to 691 a day. The site is appearing far more and ranking better on its best searches. The average just cannot show you both at once.

Why this matters to you: if your average position gets worse while your impressions climb, that is usually growth, not damage. Check which searches are new before you change anything.

Two weeks ago I told you robots were chasing filter button combinations across our city food and nightlife guides, and that one of those sites had about 17 real pages and more than 82,000 possible combinations. I also said I could not prove where the visits were going.

Now we can, and the answer is more embarrassing and much more useful.

On October 14, one robot made 92,405 visits to a single one of our guide sites in one day. Those visits hit 91,219 different web addresses. Almost no repeats. It was exploring.

So we took a sample of 4,000 of those addresses and pulled out every filter setting in them. We got 131 different ones. Then we loaded our own page in a browser and looked at the HTML.

All 131 were sitting right there in our own links.

The robot was not guessing. It was not inventing combinations. It was clicking what we put on the page. That one page carries 137 filter links, and the filtered view it leads to carries 125 more.

A different guide site of ours makes the gap even plainer. It publishes 113 filter links while its sitemap lists 32 real addresses. It took 45,055 filter visits in two days.

We did this to ourselves.

Because the fix is easy and does not involve blocking anybody.

If the robots were composing addresses out of thin air, there would be no way to stop it except shutting the door. They are not. Take the links out of the page, keep the filters working for people, and there is nothing left to crawl.

That is the next thing we are building, and I want to be straight that we have not shipped it yet. Two weeks ago I said we would try it. We waited on purpose, because if the robots had been inventing addresses the fix could not work and we would have burned a test site finding out. Now we know. Next.

Your five minute version: open a page with filter or sort buttons. View the source. Count the links with a question mark. That is how many extra pages you are offering robots, and it is in your own HTML.

Something else we learned, and it is oddly charming. The robot does not keep coming back. It sweeps a site once and leaves.

On October 20, one of our newest guide sites got 16,323 filter visits on the single day the robot first crawled it properly. The next three days: zero. And the 30 original guide sites the robot already knew? Eighteen straight days, not one filter visit.

Why this matters to you: if you see a terrifying spike in your logs, check the dates before you panic. It may be one sweep of a site the robot just met, and it may already be over.

Across the roughly 250 sites we keep logs for, robot visits went from 304,457 in 28 days to 21,797,765 in the next 28. Seventy one times more.

OpenAI's GPTBot alone made 13.1 million of them. Amazon's robot made 5.3 million. Four robots account for 99.4 percent of the whole thing, and every other robot added together moved 1.59 times.

And our seven main sites, the ones we write on? 1,403 visits a day in the latest stretch, down from a peak of 2,684 a day in late September, still about double where they were in early September.

So the headline number went up 71 times and the sites this research is about went down a bit. Same as two weeks ago, only louder. The giant number is a measure of how many web addresses we accidentally published, not of how interesting anybody finds us.

Two weeks ago it was twelve. Now it is fifteen, 41,317 visits in 28 days. Meta's robot is 82.8 percent of that on its own, even more lopsided than last time.

Four robots that had almost vanished came back this month. DeepSeek's, xAI's, Cohere's and Anthropic's search robot all went from a couple of hundred visits to around 3,300. A brand new one showed up too, Perplexity's user-triggered fetcher, on 52 sites.

Our own search numbers here: impressions climbing back, 80 a day, then 106, then 140, then 177. Clicks cooled off though. Twelve in one week in September, one to three a week since. Best search is still "real seo" at about 23rd. I said we would publish this up or down, so there it is, mostly sideways.

We checked. Over ten days we served 14,396,236 robot visits and sent back exactly zero "please slow down" responses. More than 99 percent got a cheerful 200 OK.

Which was fine, right up until October 16, when Cloudflare announced it has started grading approved robots on two things: whether they stick to what a site's robots.txt asks over time, and whether they actually ease off after a site tells them to slow down.

You cannot grade a robot on how it handles being asked to slow down if nobody ever asks. We were handing out an A to everybody.

Six days later, on October 22, three proposals landed at the standards body that handles this: a six item list of crawler best practices led by Gary Illyes of Google, plus two rival formats for robots to publish the addresses they crawl from, one from Google in JSON and one from Anthropic in TOML. Two competing formats for the same job, filed the same day, is very much how the internet gets built.

Why this matters to you: your server can politely tell a robot to come back later. Most sites never do, so most robots have never had to prove they listen. For years "be nice to crawlers" was folklore. It is turning into something written down, and rules you can point at beat etiquette you have to guess.

SignalArchitectGroup.com, one of our seven, has basically stopped showing up in Google since mid September. One impression in five weeks. Robots are still visiting it happily, 484 last week, so nothing is broken and nothing is blocked. We do not know why Google went quiet on it. It is on the list.

And an honest warning about every number here. Our own summary tables and our own raw server logs disagree by about 11 percent on the same week for the same sites. We found that on October 16 and have not resolved it. So read these to the nearest big number, not the last digit. A site that will not tell you its own error bars is one to trust less, including this one.