When I drive them out, I use a small punch (smaller then the pin) and drive the pin out with the punch form the bottom. This stops the "mushroom" effect and it seems easier to slant the punch slightly, outward so you can hit the punch square with the hammer.
Just my two sense.
