That baked-on residue is deposits left over from lead additives. If you're spending all day every day buffing it off a few thousand valves, it might pose a risk. Masks seem to be the magic cure-all for everything, so if it makes you feel better to wear one, go ahead. The positive valve rotators don't necessarily have to ratchet, so the ones that don't turn both directions are probably just dirty. They're the same ones that are used on small block Chevy V8's, so if you choose to replace them they don't need to be specific to a 261. Usually, the valve springs used with rotators are slightly shorter than the ones used for intake valves to compensate for the increased thickness of the rotators, so don't mix up intake and exhaust springs.
Jerry