Found it! It's not the brakes; the tie rod just rubs the lip on the rim and causes resistance. I guess I'll have to buy wheels or run spacers... any other ideas?