Yes, the damper is important- - - - -it relieves torsional stresses on the crankshaft created as the cylinders fire. On a V-8, the longer the stroke, the more twist the crank gets at each firing impulse, so the 350's and other engines with a 3.48" stroke need a bigger damper. The 265's and 283's had a 3" stroke, so the damper ring could be smaller. I think a 327, which has a 3 1/4" stroke had a slightly larger damper than a 283. The small block 400 didn't have enough crankcase room for big enough counterweights to be internally balanced, so the damper and the flywheel had to be intentionally manufactured off-balance, and the 3 3/4" stroke required a really big damper ring.
Bottom line, I'd reccomend using whatever size balancer the factory engineers chose to use. It might avoid a stress fracture in a crankshaft somewhere down the road. If you're using the original type front engine mounts on a 55-up pickup, it might be necessary to trim the mounting brackets that bolt to the front of the block for damper clearance when using the 350-size damper.
Jerry