Justin,
I think that you will find the hole on the backside has a small area near the outside that is cut out slightly larger than the bore. This will have a very shallow depth but forms a ridge. The result is that the swaged material fills this in and makes it extremely difficult to drive it through.
