The Stovebolt Site Forums The Shop Area General Truck Talk Old Truck Only Search Engine..

I'm building a search engine that only indexes/searches old truck information websites.

No commercial stuff, no sales websites, just real useful information, like The Stovebolt Page.

Like the VIN Cruncher, I make stuff mostly because *I* feel it's useful, and figure if I find it useful, maybe somebody else will, too.

Google is a powerful tool, but it includes alot of garbage to weed thru.. by building a search engine that is topic specific, and that includes only "human reviewed" sources, I think I can come up with a top notch search engine to find whatever information i'm looking for.. so long as it's information regarding old trucks!

If you know of a message board or general website that contains good truck info, regardless of make, please let me know, so I can add it to the spiders list of places to crawl.

Fumblin46!

This site may be of use to you. I have another site of Chevrolet from 1915-1951 but it's in a backup file on my computer and I can't find the http:// site right now, but I'll keep looking.

http://clubs.hemmings.com/clubsites/chevytalk/canadiangmc.html

P.S. I did find a more useful site after posting the above! Hope it's useful.

Cheers, George

http://clubs.hemmings.com/clubsites/chevytalk/index1.html

oldgmctrucks.com is a valuable resource, however you may not be able to link due to the sign-in procedure. This could be an issue with other sites as well.

Jarge.. any site with info about any old truck is useful! i'll be including your submissions for the beta test.

k10.. i'm going to check out that site shortly and see how "spider friendly" it is.

This project isnt so much a collection of links as it is a search engine.. just like google.. but instead of searching all of the internet, it searches only old truck specific resources.

The sites linked here on my trucks page:

http://woogeroo.home.mindspring.com/autolinks/trucks.html

Will prolly fit well in your search engine. Check them out... specifically the ones under 'resource' which is what I refer to places like the stovebotl page that have articles and forums.

feel free to snag any other links out of there as well:

http://woogeroo.home.mindspring.com

That basically the idea behind my links page... an old fashioned index... since I didn't have the comp. skills for makin' a search engine.

good luck...

-W

Thanks, Woogeroo!

The beast is built, online, and currently crawling around the web indexing pages and in general, doing it's thing.

I'm looking at the main page as it's gathering information and clicking the "refresh" button to watch the number of indexed pages grow.. as of this post, the current number of pages indexed is 723.

The longer it crawls, and the more links I feed it, the higher that number will grow. I'd like to see it over 2,000 by the end of the weekend.

For anybody intrested in checking it out, it's located here:

http://www.oldtruckstuff.com

Comments, suggestions, ideas, and additional URLs to index are greatly appreciated!

That's Great! Good work bro..Steve

Fumblin46 -

I'd highly recommend the ChevyDuty Forum... tons of 1st class vintage truck info.

http://www.chevyduty.com/frame-forum.html

So far in my search, the Chevy resources FAR outnumber the other truck resources, at least for the old stuff... lucky for me Chevy is my favorite truck!

As of this post, i've indexed over 1800 pages of old truck resources. ChevyDuty is on the "to crawl" list as I type this message.

see if there's any here of use you don't have, Fumblin, including forums - note that forums that require logins usually don't like spiders other than their own

Bill

My spider, as with all legit spiders, looks for a file called "robots.txt" off the main directory of any website it attempts to "crawl".

If a webmaster dosen't want a search engine to index specific directories, they simply add those directories to their "robots.txt" file, and the spider will not "crawl" them.

In short, if a webmaster dosen't want a search engine to index their content, all they have to do is tell it so in "robots.txt"

I don't know if having a "robots.txt" file is some kind of secret or what.. but very few webpages i've crawled so far have one.

Stovebolt.com has one, and thats why the forum content from stovebolt.com is not included in OldTruckStuff.com's indexing.

I'll go down your trucks list tonight before I go to bed and add any of the pages I don't already have indexed.. thanks!

Yea, I accidentally deleted the robots.txt file once from our server. When I finally "fessed up" to Paul Schmehl ... he had a back-up to stick back on there.

That was not real smart (of me)! That was VERY smart of Paul.

I wonder how I can get the spiders to also search the forums when they Google our site? Frank?

Peg

"robots.txt" tells a spider what directories its not allowed to crawl.

The message board here on the stovebolt is located at:

http://www.stovebolt.com/bboard/cgi-bin/

..and in your Robots.txt, that just happens to be one of the directories that search engines are told they're not allowed to crawl.

To get spiders to crawl the forms, all you'd have to do is edit Robots.txt and remove the following line:

Disallow: /html/bboard/cgi-bin

I know on the robots.txt Fumblin - I think most don't have it cuz they don't care, also you'll find more metas that just say 'index,follow'

Peg - you'd hafta talk Paul into removing the "disallow" for the relevant /bboard folders, but it doesn't really help to have the forum threads listed on general search engines when J.Q. Public can't get in w/o registering anyway

Bill

red58 has a point.. if you won't let visitors (non registered people) view/read the threads, then letting a spider crawl them dosen't make alot of sence.

If it were me, i'd set permissions on a few areas (like General Discussion) to allow visitors to be able to read them, and make those "crawl-able".

You want to encourage new people to find the board and sign up, and a good way to do that is make areas like the General Discussion searchable.. that way every word in every post in that area becomes a keyword.

When a searcher finds a link to a thread in the General area, they get to looking around, then think "Hey.. this really is a cool place, I should sign up and ask my question".

In the past i've signed up to numerous message boards that wouldn't let you read any content untill you signed up.. only to find out that there isn't any content in there worth signing up for!

As a result, most times I come across a message board that requires signup before I can read whats going on, I simply don't bother.

On signup your asking folks to provide their eMail address and name without even letting them get a look at what they're signing up for.. IMO, you outta let them at least know the juice is going to be worth the squeeze.

I'm sure there are more than 8,000 people in the world with both a stovebolt and internet access.. how many of them have found this site but never bothered to sign up because they didn't want to give up their eMail address without knowing first that the message board was cool?

How many of them have searched Google for common keywords found in the General Discussion board, but never found that thread because it isn't searchable?

I don't advocate making ALL the threads readable by visitors, because you have to have something to encourage them to sign up and be a contributing member of the board, instead of being a "lurker".. but I guarentee making the General Discussion board readable by visitors would increase traffic.. and the more traffic you have, the more clicks on that pretty JC Whitney Ad you'll get.

"let them at least know the juice is going to be worth the squeeze"

I resemble that remark!!
around here that's as easy as it is at my place - anybody w/ enough sense to have a stovebolt should know 'the apple don't fall far from the tree'

the public have full access to the rest of the site, which includes the ever growing tech tips, the feature stories, the links page and particularly the gallery, which must by now be the biggest collection of pre-72 GM trucks around - all a pretty good 'hook' for the forum, and prolly containing the majority of keywords neophytes would use

I'd think it could be a bit problematic to allow entry to just one section, and the rest require signup - having the whole forum open to viewing only w/ posting requireing signup, useta be the case, and maybe the owners would reconsider that - you may not be familiar w/ the history of the beast, Fumblin, but there are reasons ... related to 'fire prevention'

has there been an increase or decrease in signups since the log-in became mandatory? seems there's been an increase in first time posters putting stuff in the swap since the change, so I think the SBP rep has already traveled pretty far and wide

Bill

To tell you all the truth, this place is incredibly busy. I have a more emails coming in than ever. I spend a lot of time answering them or forwarding them to our volunteers.

I still can't get over how many gallery additions keep coming in. We have a lot of traffic on the site and a lot of good things happening.

I'm sure it could be more ... but I just couldn't handle it.

Making the board "spider-able" I think would be a good idea. And if they don't want to register, that's okay. I've been in the same boat when I've been searching for something. For me, it depends on how much fluff is in the rest of the site. Usually, I go for the easy ones. We do have a lot of information on the rest of the site ... and we'll even have more.

And the ads ... I'm beginning to think they are a waste of time. The only one that seems to help with the costs have been those Google ads. So, that's okay. It'll be one less thing for me to fool around with!

I think one thing about our site, too ... we are somewhat an oddity. We are not a "club" and we are not just a "web site." We definitely have a Bolt crew and they are the nuts and bolts of this place.

It reminds me of our neighbor. He makes furniture - beautiful furniture. People will wait a year to get something. It's incredibly crafted and reasonably priced. Plus, Denis is a great guy! He doesn't advertise and he is not pressured about getting more done. He will never be out of work.

I'm not saying that we are so great around here. We are doing the best we can with the resources we have. Everyone who contributes to the forums, the tech tips, features, etc. - that really helps. I have really felt the squeeze (sorry Bill) of too much happening. And I don't want this to stress me out.

You all are great. The spider thing -- I'll need to make another virtual sticky note.

I'm heading back outdoors. John just completed the last of the tree trimmings for the season! We got done in July (just in the nick) as we had hoped for! Now ... more watering. Man, we need rain.

Peg

The only reason I can come up with for not letting a spider crawl your free, non adult, message fourms would be bandwidth considerations, or your trying to block a certian IP from even viewing the messages.

The message board here at the stovebolt contains tons of good information, and grows daily. It never has to be re-formatted, it dosen't require any third party to manually compile data, with every new question asked, a new searchable keyword or topic is added.

Many search engines are wise to "meta tag fluffing" and base their rating on how frequently your search words are used in actual text.. not hidden in code META or KEYWOARD tags.

In fact, alot of search engines actually ignore everything past the first few META words, because they want to bring their searchers VIABLE CONENT THAT ACTUALLY EXSISTS, not what the programmer stuffs in the tags, that may or may not be there.

I just think it would be cool to be able to search for "How do I change 50 chevy clutch" and have it point to a particular thread that either explains it, or has links to specific resources.. NOT give me 3,000 pages of places who wanna sell me blury repro shop manuals or sell me the clutch.

Peg asked "I wonder how I can get the spiders to also search the forums when they Google our site?" and I answered.

As to weither or not it's a good idea to do that, or if there are specific reasons why all the fourms here are "reg users only", I dunno.

My search engine has indexed 2744 old truck specific pages and countin' without 'em.

Fumblin'

What a great idea! I'll have to use your engine for my '59 F*rd project!

Here are a couple suggestions:

Inliners International: http://www.inliners.org/
American Truck Historical Society: http://www.aths.org/

We're going to see if we can get this to work.

I strongly agree - getting people to search the site (the entire site) would be a big help. And hopefully, the folks who have been here a while, will get use to also searching the site and when they answer a question from a newbie, they can include some page info. I noticed WE B Old did that this morning - that's very *encouraging* to all those who submit information. I notice others do it regularly, too. Very good!

Peg

There's nothing in /html/bboard/cgi-bin/ that your spider needs to see. All the Forum posts are in /html/bboard/ubb/ and that's not blocked by robots.txt.

Thanks for the suggestions!

I added a box on the main page that lists all the websites the engine is currently indexing.. if anyone has a suggestion that isn't on the list, please let me know!

That's a decision for John and Peg to make. Spiders can crawl the Forums just fine. If they cache the pages, they can also display them without requiring registration. If you want to view the pages in the Forum, then you have to register first.

If John and Peg want to change that, it's easy enough to do - just uncheck one box. They had good reasons for requiring registration, however, so I'll leave it up to them to decide if the reasons have changed enough to justify opening it up.

I'm looking at the url for this thread as I read it in my browsers address bar, it reads:

http://www.stovebolt.com/bboard/cgi-bin//ultimatebb.cgi?ubb=get_topic&f=22&t=002614#000021

So without using /html/bboard/cgi-bin/ how would I direct link to this thread?

I thought maybe it was just my engine, but if you use google to search stovebolt.com, you can't get any results from threads, either.

A Google of Stovebolt.com for "VIN Cruncher" brings up two hits.. both in the newsletter section, (june and july) not in the threads.

The fact that the results are from June and Julys newsletter suggest that the page has been crawled by Google at least as recently as the end of June.. so why didn't it pick up any of the dozen or so refrences to the VIN Cruncher in as many threads that have been there longer?

In fact, over a dozen searches for words like "Clutch" and "chevy" result in numerous hits, all in the newsletter or tech tips section, nothing in the fourms.

I really don't know squat about UBB's directory structure or how it's run.. i've actually been contemplating setting up a UBB board just to test different methods of crawling its message board contents without allowing crawl access to /html/bboard/cgi-bin/

Perhaps its simply because the threads can't be read by a spider since registration is required? Other boards set up that way get crawled, but i've not tried it with UBB.

I just did a search on OldTruckStuff.com for UBB, in the hopes to find a UBB board, and I did.

Inliners International uses UBB, (search for Inliners UBB to see a pile o' results) and their path is

http://www.inliners.org/cgi-bin/ultimatebb.cgi?ubb=get_topic

They're using UBB Classic 6.7.0.. what I would guess is a little older than Stovebolts 6.7.2 so maybe that has something to do with it.

At any rate, it's up to Peg and John to decide if they even want the board to be crawlable, and if they do.. I don't know how to do it, and apparently neither does Google.

I see it the same Fumblin, it's a matter of the disallows

Paul - seems there has to be something on the disallow list that stops spiders from indexing the forum, because as Fumblin says, it just doesn't happen, and other 'registration required' sites do get indexed, the person just can't get directly into them w/o being registered, as you say, only into that cached page

isn't /html/bboard/ubb/ below or within /html/bboard/cgi-bin/ ?
every post address here starts with /bboard/cgi-bin// so I would expect the /bboard/ to stop the spider with out going further

I don't see a problem w/ reg to generally view the forums, it can still be required for posting

Bill

On alot of servers, .cgi files have to exsist within a cgi-bin folder in order to function (unlike PHP files that can be anywhere)

The actual data for the fourms could be stored in another directory, like /html/bboard/ubb/ but that could be just the raw data, that only makes sence when used in conjunction with the .cgi files used to dynamically generate the pages.

So while data may exsist in /html/bboard/ubb/ a web browser or a spider isn't able to get anything crawlable or readable out of it.

Hmmm....you don't have to have cgi scripts in cgi-bin if you know how to configure the server. In fact, if you'll look at the URLs for the Forum, you'll see that none of them are in cgi-bin.

Cgi-bin would be a URL like this:
http://www.stovebolt.com/cgi-bin/blah.pl or whatever.

URLs for the Forum are like this:
http://www.stovebolt.com/bboard/cgi-bin

See the difference?

Anyway, your spider probably needs access to /html/bboard/cgi-bin/ultimatebb.cgi so it can parse the files. I'll have to see if there's a way to allow that without allowing access to other stuff in there that I don't want spidered (the Member list for one thing. The last thing we need is providing spiders with everyone's email addresses and passwords!)

Try it now.

She's chewin' away on it as I type this! Thanks!

Fumblin-46

Here is one to add to the list www.brads41-46chevys.com

Hey all. Just talked to Paul. He fixed the whizzy dingy and after Google re-does something on their end, the forum listings should come up in a Stovebolt Google search (from the menu bar). Ye ha! Perfect timing you guys. I was going to re-do the Stovebolt Google page.

So ... give it a week or ten days Paul said, and we should see a difference.

Then, start using it. Try to get some of the other Bolters use to using it. We've been asking for to put more information in their gallery submissions and we are trying to stick with keywords that would help in really FINDING stuff.

So, the more folks get use to it, the easier it will be for people to find stuff. And stuff that ain't found, well ... we can just get on with that stuff then!

Yee ha!

Peg

As of 8-1, my spider still can't index the message fourms.. why, I don't know since it does other UBB boards alright.. hopefully Google will have better luck.

Fumblin,

Paul said it will take a while for Google to send out it's "stuff" ???

We'll keep checking. Once your VIN Cruncher shows up in other than those two spots, we'll know we have arrived.

Peg

Yea, i'm sure Google dosen't index pages every day, so on it's next index we'll see if it grabs fourm content.

My OldTruckStuff.com search engine dosen't have anything to do with Google, and I "crawled" Stovebolt.com as soon as that directory was available, and still can't get any results from threads.

It will be intresting to see how Google fairs on it's next update with it.

Wanna 'nother cool project? Send me an email. But I won't be back in here until the morning.

We're heading for the trees!

Peg