The collar has a couple washers and a cork seal inside, you may have to dig those out to get the ball housing off. It won't hurt to use a prybar to push it back, but be sure the tube is cleaned of accumulated gunk before trying that and use a block of wood to pad the front of the ball housing so you don't bend the edge.
