Hy che. unfotunately you will have to remove the balancer, it sounds like the balancers bore is worn oversize. You will have to look for a used or rebuilt balancer. The crankshaft was never drilled for a balancer retaining bolt so the press (interference) fit is the only thing holding it in place. I always drill the crankshaft on my engine rebuilds, hope that helps.
