If you have the spindles removed, use a drill press to do the hole enlarging. One trick I use routinely is to grind the tip of a drill bit I use to enlarge a hole down until it's a slip fit in the smaller diameter and let the tip of the bit center in the smaller hole. Reducing the diameter of about the first 1/2" of the bit is plenty. It doesn't need to be a perfect fit- - - -just enough to keep the bit from hanging up and/or wandering as it starts to cut. You can resharpen the full diameter cutting edge with a Dremel tool.
Jerry

"It is better to be silent and be thought a fool than to speak and eliminate all doubt!" - Abraham Lincoln
Cringe and wail in fear, Eloi- - - - -we Morlocks are on the hunt!
There is nothing noble in being superior to your fellow man; true nobility is being superior to your former self. - Ernest Hemingway
Love your enemies and drive 'em nuts!