The vacuum line to the manifold should have a one-way check valve that holds vacuum pressure for the brakes. Sounds like that may be leaking down. I'd start by replacing that. They usually are just pushed into the rubber vaccum line, you can feel it if you start squeezing the rubber tube...
