I didn't thank you properly 3B for you advice. Sometimes the difference between knowing what you should do and doing it, is a swift kick in the pants.

The backing plate was indeed rusted into place. Should have come off much easier. I was able to wedge some small screwdrivers in behind the race and work it far enough out to put a puller on it. The shoulder was gouged pretty bad but cleaned up good.
I never did look to see if the backing plate could have gone over the race.
Anyways thanks for the help.