have you sorted thru the wiring to make it all correct with the diagram? the fuse panel [which would be factory only if more than 1 electrical option was also installed] should be connected to the red wire on the ign sw, which should be hot all the time as it also goes to the ammeter, the light switch, and the battery terminal on the starter [your lights do work without having the key on? oh, yes, that was the original problem]
you are no doubt measuring a voltage through something else when the key is off, 3-4V is not a possibility on a properly wired system, it's 12V or no volts - which is to say something is wired wrong, sort it all out - back to the top, the violet wire needs to be connected properly to the side terminal on the starter, it provides a full 12V to the coil when you press the starter, bypassing the resistor .... a red wire should be battery voltage to the switch, ammeter and lights - revisit your wiring diagram, it shows colors .... and the 57's were all originally 12V
Bill