If you go up on the grade of a bolt, also be sure to increase the torque you use to tighten them. Grade 8 bolts tightened to Grade 5 torque specs will back out due to insufficient stretch to keep them tight.
I'm not a mechanical engineer, but as I think back a couple of eons ago to the materials class I once had to take, a steel grade 5 and grade 8 bolt should have the same modulus of elasticity - i.e. the strain on the bolt for a given stress is the same as long as no permanent deformation occurs (fastener is said to be in the "elastic" region of the stress/strain curve). Therefore, either bolt will "stretch" the same amount for a given load until the point it is permanently deformed.
A grade 8 bolt is considered stronger, since it takes more stress to permanently deform it (fastener yields and enters the "plastic" region of the stress/strain curve).
I also remember sitting through a presentation once by some fastener engineering guys. I seem to recall them saying that for most joints, the recommended fastener torque is typically a function of the desired clamp load, which in turn drives the fastener selection. (For example, an interfaces that requires a clamp load which exceeds the capability of a grade 5 fastener, would require a grade 8 fastener.) Therefore, I would be hesitant to change the recommended torque of a joint without good reason.
All the above, of course, assumes we are not talking about a torque to yield design, which I doubt is the case here.
Hopefully, some ME is reading this, and can spank me and set me on the straight and narrow if I have run off the road here...
Best Regards...
Edit: After rereading your post, I can't help but wonder if the issue of the grade 8 bolts backing out where the grade 5 bolts do not could maybe be related to the softer grade 5 bolt head "catching" on a surface, when the harder grade 8 bolt does not ?
