Mark,
Is this for your 1952 Canadian Chevy 1300???
It will help folks to know what you are working on to offer the best helpful info.
If the answer to my above question is yes, have you checked the Shop Manual for procedures?
In answer to your questions:
No, there are no retaining rings for your caps. I believe retaining rings only applied to some of the larger commercial sized trucks. Another Bolter with more info will come along and hopefully clear that point up. Yours are in fact concave caps (at least they started life that way) that are secured in the opening by dimpling the center to expand the cap and lock it into the spindle. Kind of the same process as installing engine block "flat style" core plugs.
Removal is usually done as you mentioned by drilling a small hole on the cap and then prying it out. I have used a large sheet metal screw inserted into the drilled hole and grabbed by a pair of vise-grips to pry them out. Works better (IMO) than trying to pry out with a punch.
Hope this is helpful.

Dan