2007-03-05 23:59:59 218.104.*.* GET /type/l-haikou-98.html - 80 - 202.160.179.90 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:06 218.104.*.* GET /lend/721.html - 80 - 202.160.178.131 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:09 218.104.*.* GET /lend/10172.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+http://www.yodao.com/help/webmaster/spider/;+) 200 0 0
2007-03-06 00:00:16 218.104.*.* GET /lend/6921.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+http://www.yodao.com/help/webmaster/spider/;+) 200 0 0
2007-03-06 00:00:17 218.104.*.* HEAD /lend/27870.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:17 218.104.*.* GET /smalltype/s-xiamen-4-22.html - 80 - 202.160.178.190 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:19 218.104.*.* GET /lend/9956.html - 80 - 202.160.178.92 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:20 218.104.*.* HEAD /type/s-nanjing-146.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:25 218.104.*.* GET /lend/30849.html - 80 - 66.249.72.101 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) 200 0 0
2007-03-06 00:00:26 218.104.*.* GET /lend/9794.html - 80 - 202.160.179.132 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:30 218.104.*.* GET /lend/9184.html - 80 - 202.160.178.91 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:31 218.104.*.* GET /sale/28890.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+http://www.yodao.com/help/webmaster/spider/;+) 200 0 0
2007-03-06 00:00:32 218.104.*.* HEAD /sale/6169.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:34 218.104.*.* GET /smalltype/l-nanjing-1-27.html - 80 - 202.160.179.121 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:34 218.104.*.* HEAD /sale/4886.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:35 218.104.*.* GET /lend/9986.html - 80 - 202.160.178.230 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:37 218.104.*.* HEAD /lend/1418.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:40 218.104.*.* HEAD /sale/1331.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:43 218.104.*.* HEAD /lend/347.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:43 218.104.*.* GET /smalltype/l-guangzhou-2-9.html - 80 - 202.160.178.150 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:44 218.104.*.* GET /lend/2076.html - 80 - 202.160.180.65 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+http://misc.yahoo.com.cn/help.html) 200 0 0
2007-03-06 00:00:45 218.104.*.* HEAD /lend/516.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:47 218.104.*.* HEAD /lend/8841.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:49 218.104.*.* HEAD /lend/12985.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:52 218.104.*.* GET /sale/12149.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+http://www.yodao.com/help/webmaster/spider/;+) 200 0 0
2007-03-06 00:00:53 218.104.*.* HEAD /sale/31397.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
2007-03-06 00:00:55 218.104.*.* HEAD /lend/32143.html - 80 - 202.108.22.142 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0
这是早上零点1分钟内的蜘蛛来访记录,从里面我们可以看出一下几个问题:
1.百度仍是国内最大的蜘蛛来源,也是国内抓取页面最多的搜索引擎.(12次)
2.google抓取页面的频率相对比较低(1次,在零点这一小时内最多的一分钟(0点9分)抓了6次)
3.yahoo抓取页面也很多,基本和百度持平(10次)
4.出现了一个相对较新的蜘蛛YodaoBot(以前把它当作yahoo了),抓取频率也较高
5.没有出现iask和TencentTraveler,他们抓取频率相对较低(在零点开始的前2个小时,iask只抓取了十几次,TencentTraveler抓取了一百多次),这也和他们的市场份额相符
6.所有蜘蛛都抓取js页面,不过yahoo好像喜欢js文件,基本见一个抓一个,百度相对抓的少一些.
7.百度和yahoo对参数类型的页面也是照抓不误,google好像不是很感冒.
(数据按照早上零点到5点35分1万零18次的抓取数据分析)