Will RSS clog the web?

I’m in favor of promoting the use of RSS in many aspects of information management. However, a recent wired article asks: will RSS clog the web? I’m not worried much. Why?

1. High traffic sites like slashdot are already protecting themselves. I was testing my RSS aggregator, and hit slashdot’s RSS feed several times in a minute. I was surprised to get back a message to the effect of ‘You’ve hit Slashdot too many times in the last day. Please refrain from hitting the site more than once an hour’ (not the exact wording, and I can’t seem to get the error message now). It makes perfect sense for them to throttle down the hits from programs–they aren’t getting the same amount of ad revenue from RSS readers.

2. The Wired article makes reference to “many bloggers” who put most of their entries’ content in their RSS feed, which “allow[s] users to read … entries in whole without visiting” the original site. This is a bit of a straw man. If you’re having bandwidth issues because of automated requests, decrease the size of the file that’s being requested by not putting every entry into your RSS feed.

3. The article also mentions polling frequency–30 minutes or less. I too used to poll at roughly this frequency–every hour, on the 44 minute mark. Then, it struck me–I usually read my feeds once, or maybe twice, a day. And rarely do I read any articles between midnight and 8am. I tweaked my aggregator to check for new entries every three hours between 8am and midnight. There’s no reason to do otherwise with the news stories and blog entries that are most of the current RSS content. Now, if you’re using RSS to get stock prices, then you’ll probably want more frequent updates. Hopefully, your aggregator allows different frequencies for updating; Newsgator 1.1 does.

This comes back to the old push vs. pull debate. I like RSS because I don’t have to give out me email address (or update it, or deal with the unwanted newsletters in my inbox) and because it lets me automatically keep track of what people are saying. I think there’s definitely room for abuse with RSS spiders, just like with any other automated system; after all “a computer lets you make more mistakes faster than any invention in human history — with the possible exceptions of hand guns and tequila.”. I don’t think RSS will clog the web–it’s just going through some growing pains.