Maybe you need to apply a bit more pressure on the drill bit. If it just skates on the metal, it's just going to get hot and dull.
You may want to try an expandable reamer. But you still have to go real slow with those. Shave off a couple thousandths at a time till you get to the right diameter and pin fit.
