You are describing an electrically powered device that is activated when the engine is started but is still powered when you turn the key off.
Check every hot wire that doesn't go through the ignition switch.
What comes to mind is an old diaphragm electric fuel pump. But you never know what strange thing a PO wired in with direct power.
