If the 11/32 cut true, the .005 clearance will not produce a noticeable "wiggle" over a 2"+ deep hole.
The drill may have cut oversize. I have seen this so much over the years that I always drill a test hole of the same depth in a scrap piece when I'm doing a precision fit. If the drill is a little dull on one side more than the other or is sharpened incorrectly it will cut a bigger hole. If the bit was slightly bent it will do the same.
Also the part really has to be securely clamped in the drill press vice, perpendicular to the travel of the drill. Even a slight angle will cause the drill to deflect as it tries to follow the hole. i also like to let the vice float so the bit self centers.
I just did a set of hinges for the 48 using the same pin set.
When I tested my 11/32 drill I found it cut several thousands over. Rolling the bit across a surface plate I could see a slight bend. Probably happened the last time I used it in a hand drill.
The new drill was right on.
BTW, when opening a hole like that and not doing the last cut with a reamer, I always run the next smaller drill through first so the final cut doesn't have to work too hard.
