Best guess? The gear on the shaft and the gear machined on the cam have some combination of out of round. The forces are not great. You drive the pump and turn the dizzy shaft. You might even have some side to side slop in the oil pump. Add it all up and a "loose running fit" as I was taught and...a little wobble. Some more than others. If the pump is pumping, and the timing is stable, it would seem to matter little. I suspect the loose fit in the block is the real culprit.
