I still can't grasp the problem. You say "when both power sources hit the relay". You should only have one power source at a time, either left or right. Either one should switch the relays for that side. The flasher has nothing to do with switching.
Are you sure you don't have some wires crossed. I can find no reason why the system shouldn't work if wired per the diagram.
