Brake dust is my vote. Also they could be picking up some road dust. Do you routinely drive on gravel roads? Are your backing plates in good condition, or are they damaged so as to allow dirt to get in? I would pull the drums, clean up, scuff the drums up with emery cloth and scuff the shoes as well, or if they are badly glazed run a power wire brush over the contact surfaces. That'll do it.
