Put a 1/2" fine thread bolt in that hole to give you something to grab the axle by. Give it a few quick pulls and it will come right out. Sometimes there's a slight bur on the end of the drum or the gasket it still stuck onto the drum, but it will slide right out, no retainer,
Scott