Here’s a fascinating case study of the evolution of a (large!) log analysis toolset. It follows Rackspace as they strive to allow searching logfiles from an ever growing number of servers, starting from manual ssh/grep, through various incarnations of a MySQL database, and ending up using Hadoop/Lucene/Solr. There’s (some) more info on the rackspace blog. Cool stuff.
Via George Fairbanks.
[tags]lucene, solr,case study,hadoop[/tags]
The website http://highscalability.com is great. They have the original writeup on this example: highscalability.com article
Thanks George, lots of great numbers in the highscalability post.