Just because I can't see from this one picture, did you remove the exterior key lock mechanism?
If every other pieces of hardware securing the latch to the door is removed, from what can be seen in this picture, that's the only thing I can think of to possibly keep it from coming out.
