When setting up the torque tube with shims do you do that with or without the gasket on the ball?

Thanks