The only reason I can come up with for not letting a spider crawl your free, non adult, message fourms would be bandwidth considerations, or your trying to block a certian IP from even viewing the messages.
The message board here at the stovebolt contains tons of good information, and grows daily. It never has to be re-formatted, it dosen't require any third party to manually compile data, with every new question asked, a new searchable keyword or topic is added.
Many search engines are wise to "meta tag fluffing" and base their rating on how frequently your search words are used in actual text.. not hidden in code META or KEYWOARD tags.
In fact, alot of search engines actually ignore everything past the first few META words, because they want to bring their searchers VIABLE CONENT THAT ACTUALLY EXSISTS, not what the programmer stuffs in the tags, that may or may not be there.
I just think it would be cool to be able to search for "How do I change 50 chevy clutch" and have it point to a particular thread that either explains it, or has links to specific resources.. NOT give me 3,000 pages of places who wanna sell me blury repro shop manuals or sell me the clutch.
Peg asked "I wonder how I can get the spiders to also search the forums when they Google our site?" and I answered.
As to weither or not it's a good idea to do that, or if there are specific reasons why all the fourms here are "reg users only", I dunno.
My search engine has indexed 2744 old truck specific pages and countin' without 'em.
