It is normal to see about 1/8" of the tapered face of the retaining ring in the holes.
The set screws should be snug.
I question why when you tighten them tighter you would experience any bind? That does not sound good. The retainer is only putting pressure against the outer race of the front pinion bearing, keeping it seated in the housing.
