SEO Life This Week
SEO Life This Week: One Bot Wore Fifteen AI Crawler Names
The Real SEO™ homepage held page one for four of seven measured days. The bigger lesson came from the server logs: one actor wore fifteen crawler names, and one of our own tests fell apart when we checked the weeks inside the window.
Audio Summary
Listen to This Week's Summary
This three-minute recording covers the Real SEO™ checkpoint and the crawler identity problem. It also explains the failed preference test and real-estate recovery.
Read by Kodi, AI co-architect
Read transcript
This week in SEO Life, the biggest lesson was simple: verify what the data is actually telling you.
Real SEO continued making progress.
The homepage appeared in Google's top 10 for the exact search Real SEO on four of the seven measured days. Clicks also improved, and the broader Real SEO search group moved to a better average position.
That is a meaningful checkpoint, but it is not ownership of the search result yet.
And importantly, most of that improvement began before the newest article experiment started, so we cannot honestly give the new article credit for the movement.
The more interesting discovery came from the server logs.
We found 15 different addresses presenting themselves as 9, 10, or even 11 different AI crawler identities on the same day.
Some of those requests were not normal page retrievals at all. Ninety-one were DELETE requests.
The lesson is that a crawler name is only a claim.
If something calls itself GPTBot, ClaudeBot, Google, or another crawler, that does not automatically prove who is actually making the request.
From now on, identity needs to be verified using network ownership, behavior, frequency, and eventually signed crawler authentication.
Another experiment also failed this week, and that failure was useful.
We had been testing whether individual AI crawlers consistently preferred one website over another inside a group of related properties.
The answer was no.
Four of the seven groups changed leaders.
What looked like crawler preference was really something simpler.
When one site had a very large lead, it usually stayed ahead. When the difference between sites was small, the rankings moved around.
So the original crawler preference theory was closed as falsified, rather than trying to force the data to support it.
We also finally documented the original cause of the Real SEO real estate visibility collapse.
Robots directives had accidentally told search systems not to index or follow parts of the real estate section.
Sitewide impressions dropped from more than 20,000 in March to just 437 in April.
Those directives were later removed.
Visibility began returning, reaching nearly 5,000 impressions in August and more than 5,000 through September 17th.
The timing strongly matches recrawling and reindexing, although the data cannot prove that every recovered impression came from that correction.
The larger lesson from the week is that the same type of mistake can appear in several different forms.
A crawler name is not necessarily a verified identity.
A rolling total is not necessarily a real trend.
And an attractive explanation is not necessarily what the evidence proves.
Sometimes the most useful result of an experiment is discovering that the original idea was wrong.
That is why the receipts matter.
The evidence has to survive the story we want to tell.
What the week says.
Crawler names are claims, a rolling window can turn one burst into a trend, and both errors disappear when you inspect the request and the weeks inside the total.
The Real SEO™ homepage passed the first checkpoint
The homepage appeared in Google's top ten for the exact query real seo on four of seven measured days from September 17 through September 23. It recorded 10 impressions and 3 clicks for that exact query while the broader Real SEO™ query group improved from a weighted average position of 20.268 to 17.539.
That is a checkpoint, not ownership. The broader group lost 12 impressions week over week, and the formal article experiment did not begin until September 23. Most of this movement started before that intervention, so it cannot be credited to the new article.
| Metric | September 10-16 | September 17-23 |
|---|---|---|
| Query-group impressions | 153 | 141 |
| Query-group clicks | 2 | 7 |
| Weighted average position | 20.268 | 17.539 |
| Homepage top-ten days for exact query | 3 | 4 |
| Homepage clicks for exact query | 2 | 3 |
The query group includes queries containing real seo. Homepage rows use the exact query real seo. Search Console data is complete through September 23.
| Request group | Requests |
|---|---|
| All declared AI crawler requests | 318,345 |
| GET requests | 317,280 |
| Non-GET requests | 1,065 |
| Non-GET requests from fifteen multi-name addresses | 902 |
| DELETE requests | 91 |
The fifteen-address sample was concentrated on September 25 and represents about 0.33 percent of total declared AI crawler requests in the measured period.
| Period | Sitewide impressions |
|---|---|
| March 2026 | 20,270 |
| April 2026 | 437 |
| July 2026 | 711 |
| August 2026 | 4,979 |
| September 1-17, 2026 | 5,221 |
March included 4,462 impressions from the real estate segment. September is a partial month through September 17. The recovery timing is consistent with recrawling and reindexing but does not establish causation by itself.
We stopped accepting the crawler's name as its identity
The server logs contained 1,065 declared AI crawler requests that did something other than retrieve a page. Of those, 902 came from fifteen addresses that each presented nine to eleven vendor identities on the same day. Ninety-one of the requests used DELETE.
We published the request pattern and the verification method in Fifteen AI Crawlers Walked Into My Server. It Was One Bot. The rule now is simple: a user-agent name tells us what the request claims to be. Network ownership, behavior, frequency, and signed identity tell us what we can verify.
Our crawler-preference test measured margin width instead
We expected each crawler to keep favoring the same property over time. Four of seven properties reshuffled. The groups that looked stable had leader ratios of at least 3.64 to 1, while every reshuffled group was at 1.71 to 1 or lower.
The test did not prove stable crawler preference. It showed that a wide margin is less likely to change rank than a narrow one. We closed the original hypothesis as falsified because keeping it open would turn a useful failure into a misleading success.
The spoof sample does not describe all AI crawler traffic
The verified multi-name sample represents about 0.33 percent of the 318,345 requests that declared an AI crawler identity during the measured period. It proves that the fifteen-address sample was not what its user-agent strings claimed.
It does not prove the other 99.67 percent was fake or verified. Signed crawler identity is the next measurement layer, not a conclusion we can claim before those headers and signatures are captured.
The real estate collapse finally has a documented cause
The Real SEO™ real estate case study had been tracking a sharp visibility loss without the original cause in the record. The rebuilt history showed robots directives that told search systems not to index or follow the real estate sections. Sitewide impressions fell from 20,270 in March to 437 in April, a 98 percent collapse.
Those directives were later removed. Impressions rose from 711 in July to 4,979 in August and 5,221 through September 17 without a content campaign during that recovery. The timing is consistent with recrawling and reindexing, but it does not prove that every recovered impression came from the correction.
Krisada's read and Kodi's AI read.
The most useful result this week is that one of our ideas broke. I would rather close a test because the evidence changed the question than keep polishing a conclusion the data does not support. The same standard applies to the crawler names and to the real estate case study: the receipt has to survive the story we want to tell.
The same mistake showed up three ways. A crawler name is not a verified identity, a rolling total is not a weekly trend, and an applicant name is not always the company behind a medical device. We need to keep the aliases, weekly slices, request behavior, and verification status or the summary becomes more confident than the evidence.
Keep up with SEO Life
Every Saturday issue freezes the data, records what changed, and names the metric worth watching next.