Enlarging holes is hard on twist drills, as the force is concentrated on the tip of the bit. If you're using a hand drill motor, you're probably binding and breaking the bit cutting edge because it's hard to keep them aligned. To successfully drill these, you probably need to clamp them in a drill press, center the existing hole, and has been said go REAL slow with plenty of lubricant.
Another option is an adjustable reamer, taking just a tiny bit off each pass, and adjusting the reamer until you get to the desired size.
