Originally posted by Fumblin46:
"robots.txt" tells a spider what directories its not allowed to crawl.
The message board here on the stovebolt is located at:
http://www.stovebolt.com/bboard/cgi-bin/
..and in your Robots.txt, that just happens to be one of the directories that search engines are told they're not allowed to crawl.
To get spiders to crawl the forms, all you'd have to do is edit Robots.txt and remove the following line:
Disallow: /html/bboard/cgi-bin There's nothing in /html/bboard/cgi-bin/ that your spider needs to see. All the Forum posts are in /html/bboard/ubb/ and that's not blocked by robots.txt.
