Trapped Some Web Scrapers
Recently, I found On Bots, an interesting project from 2006-05-03. Someone constructed 2 billion numbered web pages, arranged as a binary search tree via hyperlinks. They tracked accesses of the web pages for a year. It looks like they were interested in how much of their binary trees got explored, and in what order more than anything else.
I was inspired to start an experiment with a large number of cross-linked web pages. I’m not interested in what part of a “tree” of web pages a particular browser or IP address explores. I’m interested in getting a few crawlers stuck in a maze, and wasting a lot of their time because they’re poorly designed.
On August 1 2026, I opened a scraper junkyard to all visitors.
What I did
Instead of creating a large number of HTML files,
I wrote a PHP program,
and put in place an Apache mod_rewrite directive.
This combination creates a binary tree of numbered
HTML pages on demand.
The mod_rewrite directive hands off all URLs
matching the pattern /node_[0-9][0-9]*\.html
to the PHP program.
Based on the URL,
the PHP program decides its own “node number”,
and calculates
it’s parent number and left and right child numbers.
The PHP program writes out HTML with hyperlinks to
parent, left and right child URLs all referencing
other node numbers.
To prevent wiseacres from crawling the tree of URLs to depths that might cause arithmetic overflow, the PHP program does not create child or parent links when its number is 524289 or greater. That means by following “right child” hyperlinks, you can reach a URL for a page claiming to be Important Node 1048578. There are 1,048,578 generated HTML web pages in my experimental materials.
The PHP program generates HTML that browsers render like this:
This setup allows the appearance of serving a large number of unique web pages, without consuming much disk space. The numbering scheme allows deterministic calculation of parent and child web pages’ URLs, increasing verisimilitude of the generated pages. My hope was to entice various web crawlers and scrapers into following links up and down the tree, taking up their time, but giving them nothing in return.
I put a link on the front page of this website
with the anchor text Scraper Junkyard,
and href=/node_0.html,
which is the root of the binary tree of web pages
the parent and child links create.
Did any crawlers take the bait?
Oh, yes. Here’s a visualization of web traffic to the server for this blog.
The Y axis is a count of HTTP requests per day,
based on Apache “combined” format log files.
The server’s /etc/timezone says Etc/UTC.
It may be possible to do some rolling count of a number
of days or hours of requests to get closer to an
instantaneous rate of requests,
but that seems like more work than is necessary for understanding.
I used lines on the plot above to represent discrete counts.
Using points made it hard to see the extreme daily variance
after 2026-08-15.
Significance of dates on X axis
- 2025-03-08 - Start goofing on Meta
- 2025-06-17 - Stop goofing, start giving out 404s to Meta
- 2026-01-01 - New Year’s Day, give a visual anchor
- 2026-03-22 - Begin goofing on ChatGPT
- 2026-06-17 - .git/config spider
- 2026-08-01 - Start my own On Bots scraper junkyard experiment
- 2026-09-30 - End of the experiment
Virtually all of the rise in daily requests after 2026-08-01
is due to requests for /node_*.html URLs
from the scraper junkyard.
Here’s what web traffic to this server looked like
without the scraper junkyard requests.
Subtracting the requests for /node_*.html URLs,
my web server’s rate of requests from 2026-08-01 to 2026-09-30
is similar to the rate of requests during the period
from 2026-03-22 to 2026-07-31.
Above, the daily web traffic for scraper junkyard requests.
Scraper junkyard requests have URLs of the form /node_[0-9]+\.hmtl.
The graph represents 15,440,078 HTTP requests
from 4,248,158 unique IP addresses.
There was a maximum of 923,872 requests on 2026-09-13.
There were 210,879 requests per day median,
and mean 253,115.9 requests per day.
Who makes requests for junkyard URLs?
I looked
only at requests for URLs of the form /node_[0-9]+.html,
because those URLs are the extra load from 2026-08-01 on.
I could extract IP addresses
from the Apache “combined” format log lines.
From that, I got a list of IP addresses with counts of requests
sorted by number of requests each IP address made.
I used the sort | uniq -c | sort -k1n shell pipeline pattern.
Doing command line whois invocations on the IP addresses with the highest counts of requests gave me some of their owners: Google, Oracle, Railway, and a European ISP Contabo GmbH. I wrote a program that let me label the file with IP addresses and counts of requests by the CIDR they belong to.
I performed repeated cycles of using whois
to find CIDRs,
labeling previously unlabeled IP addresses
with owners of the CIDRs,
then finding the remaining unlabeled IP addresses
with highest count of requests.
I eventually found Hurricane Electric’s BGP Toolkit. I could look up an IP address, find its ASN, then look up routes in the ASN to find the IP address owner’s other CIDRs. I built up a file of about 3900 CIDRs. Those 3900 CIDRs let me label about 28% of the IP addresses with an ISP or some other entity’s name. The labeled IP addresses made 72% of the HTTP requests.
I could not find any good way of discovering owners of the remaining unlabeled IP addresses. I tried finding the prevalence of 3-byte prefixes of IPv4 addresses. That is, an IPv4 address like 101.165.216.113 has a prefix of 101.165.216, as would 101.165.216.1 and 101.165.216.254. I found a few 3-byte prefixes that encompassed more than one IPv4 address, but not all that many. A large proportion of the IP addresses that requested a scraper junkyard URL only requested one URL.
For clarity: I know that IP addresses are assigned to legal entities, and those entities trade and sell the addresses regularly. My file of CIDRs is only somewhat accurate, and as time passes, will become less accurate. The names I have matched with CIDRs are also kind of vague and sloppy. Here’s the file of CIDRs and entity names for your inspection.
| Entity | Count of Requests |
|---|---|
| Contabo | 6278685 |
| Unmatched | 4357504 |
| Alibaba Cloud | 3143673 |
| Rapid Seedbox Ltd | 786888 |
| Oracle | 206191 |
| 183713 | |
| Viet Nam Post and Telecom | 89562 |
| Nuba IP B.V. | 82644 |
| APPLE | 73010 |
| Railway | 59456 |
The top 9 ISPs without unmatched account for 10,903,822 of a total of 15,440,078 scraper junkyard URL requests. “Unmatched” requests are those HTTP requests made by an IP address I didn’t have the patience to dig up an ISP or other entity for.
Above, traffic visualization of the 5 most demanding identified ISPs/address space owners. IP addresses owned by Google and Oracle don’t appear prominently. Those two address space owners didn’t make that many requests compared to numbers 1 through 3. Numbers 6 through 10 don’t really show up at all.
Per ISP Numerical Characteristics
| IP addresses | HTTP Requests | Mean Reqs/IP | |
|---|---|---|---|
| Contabo | 8616 | 6278685 | 728.7 |
| Seedbox | 786870 | 786888 | 1.00 |
| Alibaba | 4233 | 3143673 | 742.7 |
It appears that the use of Rapid Seedbox IP addresses is different from the Contabo and Alibaba use.
Contabo
On the graph, “Contabo” points count requests from
IP addresses I could identify via whois and BGP
data as residing in ASNs and routes put forth by
Contabo GmbH.
I found 251 CIDRs associated with Contabo.
Contabo IP addresses made 0 (zero) requests
on 2026-09-09 and 2026-09-10.
Between 2026-09-17 and 2026-09-23,
Contabo IP addresses made only 2 requests.
It’s clear that Contabo IP addresses caused the 2026-09-02, 2026-09-07, 2026-09-13 through -15, and 2026-09-24 through -26 peaks in traffic.
I conjecture that there’s at least one cloud-based crawler operating out of Contabo. The dual peaks 2026-09-02 and 2026-09-06 suggest that maybe two or more crawlers are at work. It’s possible that the gaps in requests represent one crawler finishing, and another starting up a few days later.
I further conjecture that if there’s only one crawler operating out of Contabo IP address space, it is centrally controlled. The crawler quit making requests, actually made 0 requests, on 7 days during the experiment. There’s no way some distributed, sloppy script kiddie botnet could be that well coordinated.
Having 250+ CIDRs available means that it’s hard to see the pattern of this crawler’s requests in Apache log files without a giant pile of data.
Rapid Seedbox Ltd
“Seedbox” points are counts of requests from IP addresses in ASNs and routes published by RapidSeedbox Ltd. Rapid Seedbox offers IP address rentals, “clean, blacklist-free IPv6 subnets designed for web scraping”. They also offer Rotating Residential Proxies. I had never heard of them before I tried to identify IP addresses hitting my scraper junkyard. They seem to have a presence in 11 countries, including Seychelles where they are nominally headquartered. They seem dodgy to me. I ended up with over 900 CIDRs attached to Rapid Seedbox.
The use of 1 Rapid Seedbox IP address per HTTP request is impressive. Because Rapid Seedbox advertises its suitability for web scrapers, I conjecture that there’s a big web scraping operation working with Rapid Seedbox.
Alibaba Cloud
“Alibaba” refers to IP addresses I can attribute to Alibaba Cloud. I found 402 CIDRs for Alibaba Cloud.
I’d say that another crawler operation, separate from the Contabo and Rapid Seedbox operations, works out of Alibaba Cloud. Even though the average requests per IP address is about the same, there doesn’t appear to be any coordination between Contabo and Alibaba crawlers.
A hidden crawler
There’s at least one more crawler hiding in the data. I could not assign an owner to IP addresses making 4,357,504 (about 28%) of the HTTP requests. The Contabo crawler(s) and the Alibaba Cloud crawler(s) had one IP address making lots of requests. What does counting the requests made by unique IP addresses show? I could think of two ways to define unique in this context.
- Unique in a day
- IP address that makes a single request
The period during which “uniqueness” happens is arbitrary in both definitions. I also expected a steady trickle of unique IP addresses due to Real Humans™ visiting the scraper junkyard link, trying a few clicks, then moving along.
Unique IP address counts, either daily or globally unique, reveal that the 2026-08-22 to 2026-08-31 bout of scraper junkyard requests originated mainly from unique IP addresses. I think that indicates a crawler like the Rapid Seedbox crawler, carefully using one IP address for one request. Unlike the Rapid Seedbox crawler, this crawler used multiple ISPs, and that makes it very hard to identify as a unitary crawler.