I guess a lot of wear could produce that situation. Usually the ball will be egg shaped if it is worn and sometimes there will be a groove in it. I think if you tried replacing the other parts and it didn't do the trick then it would be time to do the balls. I believe the two spacers are different and someone before you could have put the wrong parts in there.
