Adding a USB charger is a great idea. I think these devices always draw a small amount of power even if nothing is connected to them so I would connect it to switched power instead of the always hot cigar lighter circuit.
Make sure that your USB charger puts out enough power for your device. Some USB sockets (possibly a radio's USB port) only supply 0.5A but aftermarket chargers commonly supply at least 1.0A and often 2.1A. If you're using your phone for navigation, I would consider looking for a charger with at least one port with 2.4A or, even better, a QC 3.0 USB port.
See
Quick Charge 3.0 explained: what you need to know and
Navigation.
