There's always that little bit of drag in a certain spots, even after a brake job. It's just the way it is. If you can't feel any brake shudder in the pedal or in the steering wheel, don't cut any more material from the drums, it's a waste of time and material. A slight drift when applying the brakes could be an alignment problem or a tire problem. It will take some diagnoses to get to the bottom of it. If the adjustment is close, it won't pull unless one wheel is contaminated or a different shoe lining than the other. A lot of possibilities,
Scott