A rather voracious bot which announces itself as:

Mozilla/5.0 (compatible;AspiegelBot)

and comes from a wide range of AWS IP addresses (non-exhaustive list below). It seems to be quite new and as of the time of writing there don't seem to be any meaningful search results, other than a handful of links to website statistics pages.

• 114.119.160.18
• 114.119.160.31
• 114.119.160.43
• 114.119.160.50
• 114.119.160.56
• 114.119.160.87
• 114.119.160.101
• 114.119.160.106
• 114.119.160.141
• 114.119.160.144
• 114.119.160.150
• 114.119.160.167
• 114.119.160.171
• 114.119.160.189
• 114.119.160.200
• 114.119.160.217
• 114.119.160.224
• 114.119.160.245
• 114.119.161.4
• 114.119.161.19
• 114.119.161.43
• 114.119.161.51
• 114.119.161.55
• 114.119.161.76
• 114.119.161.82
• 114.119.161.83
• 114.119.161.113
• 114.119.161.114
• 114.119.161.116
• 114.119.161.121
• 114.119.161.132
• 114.119.161.134
• 114.119.161.150
• 114.119.161.167
• 114.119.161.183
• 114.119.161.227
• 114.119.161.251
• 114.119.162.2
• 114.119.162.19
• 114.119.162.34
• 114.119.162.40
• 114.119.162.44
• 114.119.162.56
• 114.119.162.57
• 114.119.162.64
• 114.119.162.66
• 114.119.162.91
• 114.119.162.133
• 114.119.162.165
• 114.119.162.195
• 114.119.162.200
• 114.119.162.206
• 114.119.162.207
• 114.119.162.212
• 114.119.162.224
• 114.119.162.249
• 114.119.162.250
• 114.119.163.3
• 114.119.163.13
• 114.119.163.16
• 114.119.163.20
• 114.119.163.56
• 114.119.163.63
• 114.119.163.81
• 114.119.163.86
• 114.119.163.108
• 114.119.163.117
• 114.119.163.121
• 114.119.163.123
• 114.119.163.142
• 114.119.163.147
• 114.119.163.161
• 114.119.163.173
• 114.119.163.183
• 114.119.163.197
• 114.119.163.239
• 114.119.164.1
• 114.119.164.3
• 114.119.164.7
• 114.119.164.33
• 114.119.164.46
• 114.119.164.47
• 114.119.164.71
• 114.119.164.81
• 114.119.164.85
• 114.119.164.108
• 114.119.164.115
• 114.119.164.118
• 114.119.164.132
• 114.119.164.155
• 114.119.164.166
• 114.119.164.183
• 114.119.164.196
• 114.119.164.206
• 114.119.164.207
• 114.119.164.213
• 114.119.164.233
• 114.119.164.253
• 114.119.165.6
• 114.119.165.15
• 114.119.165.23
• 114.119.165.28
• 114.119.165.41
• 114.119.165.42
• 114.119.165.52
• 114.119.165.59
• 114.119.165.83
• 114.119.165.108
• 114.119.165.115
• 114.119.165.120
• 114.119.165.122
• 114.119.165.125
• 114.119.165.127
• 114.119.165.148
• 114.119.165.168
• 114.119.165.169
• 114.119.165.175
• 114.119.165.199
• 114.119.165.213
• 114.119.165.227
• 114.119.165.229
• 114.119.165.230
• 114.119.166.1
• 114.119.166.58
• 114.119.166.79
• 114.119.166.105
• 114.119.166.107
• 114.119.166.156
• 114.119.166.231
• 114.119.166.239
• 114.119.166.240
• 114.119.166.241
• 114.119.167.13
• 114.119.167.38
• 114.119.167.45
• 114.119.167.48
• 114.119.167.56
• 114.119.167.62
• 114.119.167.96
• 114.119.167.109
• 114.119.167.130
• 114.119.167.139
• 114.119.167.154
• 114.119.167.161
• 114.119.167.181
• 114.119.167.209
• 114.119.167.215
• 114.119.167.241
• 114.119.167.248

I was idly watching some Apache access logs scroll by (well actually I was busy doing something, but like to keep an eye on things to spot any interesting or worrying trends early) and noticed a bunch of entries like this:

94.228.34.238 - - [06/Feb/2011:06:21:23 +0100] "GET /blog/?o=10 HTTP/1.1" 301 290 "-" "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)"
94.228.34.238 - - [006/Feb/2011:06:21:33 +0100] "GET /blog/?o=40 HTTP/1.1" 301 290 "-" "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)"

Never noticed that UA before, and it never seemed to follow up on the (perfectly valid) 301 redirects. A scan through 3 months of logs shows it's been doing that all the time - what a dumb bot.

Checking the URL provided, this appears to be the home page of a UK-based company providing "Social Media Monitoring Tools" - and who don't have the courtesy to provide any more information about their bot / crawler. Which is evidently not popular in some quarters.

So, as "Brandwatch" provides neither myself not the sites I run with any conceivable benefit, it's on the blocklist they go.

(I wonder if they monitor their own brand?)

I noticed this fella hammering away at one of my sites:

195.42.102.25 - - [31/Aug/2010:02:16:10 +0200] "GET /some/url.html HTTP/1.1" 200 13427 "-" "Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)"
195.42.102.25 - - [31/Aug/2010:02:16:12 +0200] "GET /some/other/url.html HTTP/1.1" 200 55822 "-" "Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)"

The URL in the browser UA string appears at first glance to be some kind of search site, but the "category link" on the page are stuffed with the kind of keywords you associate with spam of all kinds and no actual search results were returned for a couple of common keywords.

It now has its own entry in the Bad Bot List and some special rules at firewall level. Hasta la vista.

Here at Penguin Blogs, Inc. we get a fair bit of comment spam. Most of it is automatically blocked by a fairly ingenious filter mechanism, but from time to time unknowns get through such as contentless posts like the following:

Very nice posting. I liked it.
thank you for your great posting.
Well that was a nice post

Purportedly this was written by a "Nick Matyas" of "Web Royalty" - what looks to be a legitimate SEO consultancy (look them up yourself, I'm not giving them the benefit of a link) but who are either using underhanded spamming methods, or have made a bad choice in outsourcing their own SEO.

Whatever, their URL is now on the filter list for this and many other sites, so they won't be troubling us here again.

Deja-vu all over again...

74.86.249.98 - - [19/Dec/2007:10:17:22 +0100] "GET /path/to/file HTTP/1.1" 200 10049 "-" "Mozilla/5.0 (compatible; Gigamega.bot/1.0; +http://www.gigamega.net/bot.html)"

38.113.234.181 - - [27/Nov/2007:13:21:24 +0100] "GET /robots.txt HTTP/1.0" 200 612 "-" "voyager-hc/1.0" 38.113.234.181 - - [27/Nov/2007:13:21:35 +0100] "GET /path/to/some/file.html HTTP/1.0" 301 363 "-" "voyager-hc/1.0"

38.113.234.181 resolves to crawl1.cosmixcorp.com, and cosmixcorp.com redirects to kosmix.com - a California, USA-based outfit which appears to be legit in a "we're a cool California start-up" kind of way. Not quite sure what they're doing (hey - it's Web 2.0), but it evidently involves crawling without an identifiable bot UA.

Our secret sauce (all Web 2.0 companies need one) is our categorization engine that crawls billions of Web pages in a unique manner to create algo-generated home pages…more on this later.

In the meantime kosmix.com has vanished from the internet - and good riddance.

A string of entries from a bot calling itself LiteFinder/1.0: never heard of it, though the URL provided (http://www.litefinder.net/about.html) does work and claims it's a "a research project started by a group of Indian candidates from the cities of Bangalore, Patna and Jaipur.".

This stupid bot doesn't understand UTF-8 encoded URLS...

220.208.55.xxx - - [03/Aug/2006:07:45:03 +0200] "GET /ã??ã?«
ã??.html HTTP/1.1" 404 2422 "-" "Pockey-GetHTML/4.14.1 (Win32; GUI; ix86)
"
220.208.55.xxx - - [03/Aug/2006:07:45:04 +0200] "GET /ã??ã?£
ã?ªã??ã?³.html HTTP/1.1" 404 2422 "-" "Pockey-Get
HTML/4.14.1 (Win32; GUI; ix86)"
220.208.55.xxx - - [03/Aug/2006:07:45:06 +0200] "GET /è??å?¤
�治�.html HTTP/1.1" 404 2422 "-"
 "Pockey-GetHTML/4.14.1 (Win32; GUI; ix86)"