IF I read correctly, my opinion is that will only work if bracket is clamped up and in; and all holes are line drilled. Otherwise the tolerances of many holes on two planes will not allow it. In other words, you can not drill holes in each piece separately if using a near tight fit fastener. Not saying that is what Jerry said.
Jaydee: Line drilling means all holes are drilled thru all pieces at once, as all pieces are clamped together. It would be near impossible to do if several holes are wallowed. You may end up with some pretty big holes/bolts. I recommend you refer to, and consider, my post 1183454.
Once you get the bracket off, if you decide to completely remove it, you will be in better shape to access the amount of hole elongation. (wallowization)

Hey, it's a word now.