It sounds like you need to get a little more aggressive than just sanding on the inside of the washer. I'd try a die grinder and take off more than just a few thousandths from the ID. I wouldn't worry about the OD of the washer. If you've flattened the washer at all it will reduce the ID a bit. When grinding on the washer, slip it over the axle to check that there's enough clearance before installing anything in the housing.
