Did you put the cork seal in the ball collar?
5.572 Seal unit/cork seal Scroll down
The cork seal keeps the ball collar from making contact with U-joint ball housing.
The cork seal may not be thick enough.

Q.Does anyone know exactly where the ball is supposed to contact the housing?
A. If you mean tranny tail housing, the u-joint ball housing swivels inside it.