I was thinking along the same lines, although it would be difficult to do with that propeller shaft in the way.
I made another tool from a piece of 1" pipe with a union on the end (see photo). The union is almost exactly the same outside diameter as the bushing. The pipe's inside diameter is only slightly larger than the propeller shaft. So, it pretty much centers itself on the bushing which puts the force on the aluminum bushing instead of the brass insert.
I was thinking that my problem might be that the wood was absorbing most of the energy of the hammer blows. It transferred the force okay until it got really tight, and then it was cushioning the blow a little too much.
