David,
What are you watching that is wobbling? The brake drum is captured with the wheel studs to the hub. I might see the periphery of the drum moving back and forth if it had been knocked off kilter with the hub. The hub itself is a casting and its rough OD may have some run out as it's machined true to ID bearing fits and all other machined surfaces including the brake drum.
Stuart