They do not come out. I made that mistake before. They are critical to the alignment of the pump shaft collar in the block. I was scolded here on Stovebolt by more than one Stovebolter for doing so.
The pump is removed by loosening a lock bolt on the shaft inside the engine and will slide down and out.
