"I forgot to add that the crank and balancer has been drilled so that the balancer can be bolted on due to the keyway being worn."
Is the worn keyway on the balancer, or the crank, or both? I would start looking there. All the bolt will do is keep the crank from falling off once the balancer has been installed with an installer tool. It may hold it on straight, or anything but straight if the balancer and/or crank snout is goobered up.
Carl