Yes, you do have to remove the distributor. The clamp that holds the distributor down and keeps it from turning is attached to the vacuum advance. The shop manual has a nice section on removing it. It tells you to turn the engine until the rotor is pointing perpendicularly out from the engine, then make a mark on the distributor housing where the rotor is. Then you can remove the distributor and know how to position the rotor when you put the distibutor back in.
MIne came out very easily. And my new vacuum advance works !!

My dad always told me, "Don't force it, get a bigger hammer".