My own experience is that I got a noticeable performance increase by changing from a 2-1/2" Flow Master muffler to a school bus muffler. I don't like the low RPM twitter but it doesn't "drum" at 2800 RPM. So in my case it seems the choke point has been the muffler. Otherwise, I have a Offy 2x1bbl manifold and a Fenton exhaust Manifold with the exhaust-heater tubes going to the intake. Works well except for carburetor icing at idle in damp weather till the thermostat opens but it did that anyway with the original manifold and working flapper.
I've made so many small modifications over the years that I can't really remember what had the most benefit. The two biggest bennies were changing from a 216 to a 235 and then to a 261, with compression going from something like 6.7:1 to about 9:1. Gas mileage improved a lot with a cam change and then with the compression boost.