Really starting to look like the exhaust. I drove it to work today, not just around the block, so it was more of a real life test than this weekend, and it is happening a little less, and is much much quieter now after tightening the bolts. I'm going to get a new gasket as Builder suggests (they're cheap) and if that doesn't work, I will try the machining suggestion from crenwelge.
Builder, just FYI the vacuum advance does not jump or move when it makes the sound, and I didn't feel anything around the edge of the heads when I blipped the throttle.
Again, thanks to ALL of you guys for the help! I love this forum.