Is your motor supposed to have a vacuum advance? Most do but I am sure there are exceptions depending on application. If your application calls for a vacuum advance, which is a separate system that accompanies the internal mechanical advance, you will want to get your hands on a replacement and install one. That could also be one of your issues moving from idle to off idle. What year Is the motor and what application are you using it in?
I don't think it originally came with one. In my manual it says that models 280-22 and up have centrifugal advance. Mine is a 1954 GMC 353-27 according to my ID plate in the door. The only odd thing is that my manual doesn't list the 350-27 as an option. It only has the 350-24. I'm not sure if the engine is original either. The engine serial number is 270I029665 if that helps at all. I think the character between the zeros is a capital i, not a one. I also replaced the fuel pump a while back and it is the type with no vacuum pump built in. As far as i can tell the only things that use vacuum are the brakes, wipers and rear axle shifter.