MemeCode - Blog

Googlebot Is Sick

Date: 25/5/2003

Yesterday (something that claimed to be) Googlebot crawled my site, looking for pages like:

iftp.php/src/src/data/&
iftp.php/src/src/data/iftp-win32
iftp.php/src/data/&

etc etc, ad infinitum. And it kept on doing it 61 thousand times!!! So my stats had 61000+ hits for yesterday (which I fixed manually). Not happy Jan! Eating my bandwidth, screwing my stats up, and loading up the server for no good reason.

So what to do, what to do?

I've put a filtering on the site that logs all attempts to get pages with ".php/" in the URL and then not give the page to the requestor. This should slow Googlebots progress down somewhat. It still should be able to crawl valid pages, just not the weirdo freaky URL's it makes up out of thin air.

From:

Email (optional):

(Will be HTML encoded to evade harvesting)

Message:

Remember username and/or email in a cookie. Notify me of new posts in this thread via email.	BBcode: [q]text[/q] [url=link]description[/url] [img]url_to_image[/img] [pre]some_code[/pre] [b]bold_text[/b]

Sitemap