Inadequate freeplay can cause this. If it is too tight at the center, then it will 'jump' off-center with any little push. Check caster, a little extra positive is not bad, except low speed steering might be heavy, but wandering will be better, and recheck freeplay. Bearing preload should have no effect on wander.
