You say you removed the set screws but if they are not all the way out then they might still be preventing the handles from coming off. If the screws are totally removed it might take some gentle force (rubber mallet) to break the hold they have on the crank axle.
