googlebot is retarded

[BOLT]

I've added tons of content lately and all it does is search the members list!

[Scott]

Use a robots.txt file (should be able to find a vB one from somewhere) to limit where it's searching - nobody will really be interested in finding a member's list via google.

[BOLT]

Use a robots.txt file (should be able to find a vB one from somewhere) to limit where it's searching - nobody will really be interested in finding a member's list via google.

I dont know how to do that. I dont even know what that means.

I wish I could just tell google to scout a specified list of threads I give it and nothing else. I've made a "best of threads" thing for it. Is this an actual person or a fully-automated machine?

[Scott]

This site (http://www.robotstxt.org/wc/robots.html) is quite good at explaining it (though there is a bit to read). Basically, a robots.txt file stops spiders from looking at pointless pages, such as member lists, post form, etc that wouldn't be very useful for someone searching.

Here (http://www.vbseo.com/f4/robots-txt-help-8608/) is an example of a robots.txt file for vBulletin.

[vtabletop]

Using a robots.txt file to exclude the part of your site that you don't want google bots to search is a good idea.You might also consider submitting a sitemap to google if you haven't already done so.The sitemap makes the navigation and searching process easier for the robots.Whenever you add new content to the site ,the bots will read and update it immediately.