Its just stuck on there from years of grease/grime. Tap it gently with a hammer from the back or use a pry bar to work it off the spindle bore.