Ottos50, 1st off, Welcome to Stovebolt!

On your question about driving out the old bushing and seal, I would suggest you read pages
4-5 and
4-6 of the 1948-51 Chevy Truck Shop Manual. Pay close attention to the paragraph titled
Propeller Shaft Bushings and Oil Seal, and "
Fig. 8" as your 1950 axle
may be one of the early design like the 1948-49. Those bushing have "dowel pins" that keep the original upper and lower bushings in place in the tube.
You must drill out those dowel pins to be able to remove the bushings and seal. You should be able to confirm if your torque tube has those dowel pins as the are obvious indentations in the tube side.
If your tube is the later interference fit bushing style, those should be able to be driven out like your original idea. Let us know what you figure out.