You are referring to the heat riser, it's best to keep it, they help in cool damp climates.
If you are mechanically inclined, you can take the flap out, with some work, and put tight fitting bushing in the manifold to help seal the leaks. The flap is welded on the shaft and requires some grinding, it would also will require some welding to put it back. It's about as much work to take it out and plug the holes as it is to fix the holes and repair it for good.
