If there's no big, heavy flakes of rust, just drive it. If you want to remove most of the rust first, use 100-grit body sanding paper, and scrub it vigorously. Yuu're not going to hurt the drum, or remove enough metal to matter. We use 100 grit paper to scuff-sand disc brake rotors after machining to get a "non-directional" finish that increases pad life.
Jerry

"It is better to be silent and be thought a fool than to speak and eliminate all doubt!" - Abraham Lincoln
Cringe and wail in fear, Eloi- - - - -we Morlocks are on the hunt!
There is nothing noble in being superior to your fellow man; true nobility is being superior to your former self. - Ernest Hemingway
Love your enemies and drive 'em nuts!