I'm betting the wheel cylinders are frozen on three of the wheels. All of your stopping power is happening with just one drum and if it's not pulling hard to one side or the other the one working is on the rear...
If I'm right, one of these time you'll step on the brakes and one of the frozen cylinders will break free just long enough to lock up that wheel.
Mike B
