If you have been using a punch and hammer after cutting off the head then you're using too small a hammer and punch. I would suggest you get a 12 or 15 inch punch and use a short handle sledge about 4 lbs. One or two licks with that and the remaining rivet will shoot across the room.
