The "adapter" is really a speedometer speed correction device to account for the two speed rear axle. It must change gears at the same time the rear end changes gears.
Don't remove the "adapter" unless you know for sure it is the problem. So far it sounds like the "adapter" is fine, and the speedometer or cable is the problem.
Grigg
