The distributor, the block, the oil pump, and the camshaft have to be in perfect alignment. Those "pins" are actually headless bolts- - - -and the factory used a precision fixture to align the bracket before the piece was bolted in, and peened from the inside.
I can walk you through removing and replacing that piece, but not with a keyboard. Drop me a PM, and I'll respond with a phone number. You'll need to have access to a machine shop, and someone skilled enough to make the alignment fixture, and it's a "one and done" operation- - - -do it right the first time, or end up with a boat anchor! I think it's possible the holes for the oil pump and distributor might have been line-reamed at the factory AFTER the bracket was installed.
Jerry