Blue Coat: If your blog is getting bandwidth exceeded messages

A program called Blue Coat is used by large corporations and governments to protect their web surfers. It looks for “link rel=’archives’” which lists posts by month, then slurps down everything it can find looking for spyware and phishing. However, it can slurp too much and too fast. My main blog, polizeros, has been getting a few bandwidth exceeded errors lately because of this. Other much larger blogs have had problems with programs like Blue Coat too, including one blog that runs on multiple servers and gets 150,000 visits a day.

If you have a WordPress.org blog, here’s the solution. Look for this line in your template.

“<?php wp_get_archives(‘type=monthly&format=link’); ?>”

Then delete it (remarking it out didn’t work for me, deleting it did.) Then Blue Coat won’t have anything to slurp on. It’ll just analyze the current page, which is fine. (wp_get_archives produces internal code that is not viewable when reading the home page but which is meant to be read by spiders. The calendar will still work fine.)

I can certainly understand that corporations and governments need to watch out for nasties on the Net, but such programs really do need to realize that slurping down everything in sight and apparently ignoring robots.txt can disable websites too. And that’s hardly fair to the websites.