Tuesday, May 16, 2006

This morning I found one of my sites had been subjected to a deep crawl by a bot naming itself "Francis/2.0 (francis@neomo.de http://www.neomo.de/)". The site seems to be an experimental but legitimate German-language search engine. The first hits from the bot were to robots.txt, the although the site's crawler information page doesn't indicate what entries it interprets, if any. Requests look like this:

85.10.204.13 - - [16/May/2006:19:19:09 +0200] "GET /robots.txt HTTP/1.1" 206 390 "-" "Francis/2.0 (francis@neomo.de http://www.neomo.de/)"
85.10.204.13 - - [16/May/2006:19:19:09 +0200] "GET /robots.txt HTTP/1.1" 206 390 "-" "Francis/2.0 (francis@neomo.de http://www.neomo.de/)"
85.10.204.13 - - [16/May/2006:19:19:24 +0200] "GET / HTTP/1.1" 206 1949 "-" "Francis/2.0 (francis@neomo.de http://www.neomo.de/)"
85.10.204.13 - - [16/May/2006:19:19:25 +0200] "GET / HTTP/1.1" 206 1949 "-" "Francis/2.0 (francis@neomo.de http://www.neomo.de/)"

Interestingly all requests returned with HTTP status 206.

Posted at 11:20 PM |Comments (0)