Dean, you should have seen the lumps on the outside of my u-joint ball housing. I filed most of them off on the outside, filled some of the gouges on the inside with weld and ground them off. I was only worried about spots where it might be too thin.
I would have hit the outside of yours with emery cloth like you did and left the inside alone. It was almost pristine before you ground on it. You're too much of a perfectionist.

I've been thinking about chucking mine in a lathe to make the ball spherical again.