As long as the lead screw on the forstner bit isn't too close to the bolt hole, it shouldn't wander. I think you could do it carefully without anything assisting, but your idea of a plywood guide would probably work well.
Canine man's suggestion of plugging the bolt hole is good as well, but I don't think you'd even need to glue the dowels in place. They'd just be enough to keep the lead screw from breaking into the bolt hole. Once the edge of the bit cuts into the wood, it should keep it from wandering as long as you keep the drill shaft perpendicular to the wood. I have a gizmo that's a drill alignment tool, similar to
THIS, that installs between the chuck and the drill motor with some slid rails that would assist in keeping the drill shaft perpendicular. You could screw the base of that to your plywood guide and be done with the whole job in 15 minutes.
