One of two things. Either a brake is grabbing on the side it pulls toward, or a brake is not working on the side it's pulling away from. One way to check is to stop on loose gravel with someone watching. If one grabs, it's easy to see. If one isn't grabbing, try a harder stop to see if one isn't working.
