Someone here on Stovebolt suggested this trick I found helpful when I had this task awhile back. They said to use a center punch and drill into the ball stud about 1/2 inch before using a drift. That was after grinding the swagged end flush first.