Brian,
Your 6V53 may have a gummy governor, I am just about finished cleaning and rebuilding the one on my 4-53T, hope to have it back on tomorrow. At least on the inline engines the top governor housing gets very little to no oil, and will condense water and get gummy and rusty, causing the symptoms you describe. (later model inline governors have an oil line going to the top to help with this problem)
Take the governor off, apart, and clean it all up and reassemble with new parts where needed, check the governor weight pins for wear as well. At the very minimum remove the idle speed adjustment cover, then with a spanner wrench remove the high speed spring retainer, and clean all the parts that fall out. Also check for the shaft in the top cover being worn, and don't forget to grease it every now and then. If it is worn best to ream it out and install a bushing (9/16" x 7/16" x 1.25"), as otherwise the new top cover is about $150, and make a new shaft, or spend $60 on a new one.
I don't think your hanging idle is an adjustment problem as much as it cold be a sticky or worn governor problem. But none the less, a proper tuneup according to the book would be an easy fix if that's all it is.
Idle speed should be 500-600, depending on application, so if you have an air compressor you want to be on the high side of that range.
The idle speed is under the small cover same as your 3-53, to adjust it the book says first back the buffer screw out about 5/8". Set the idle speed, adjust the governor gap, adjust the rack, set the max speed, reset the idle speed, and finally adjust the buffer screw. That's the readers digest version, best to get the service manual out and go over all 6 pages in the tuneup section for your governor, section 14.3.2.
Primary filter (suction side of pump) is usually on the frame by the tank in a truck application, your situation sounds normal.
Return line needs to be submerged where it enters the tank, helps to reduce fuel draining out of the system when sitting and being hard to start. Obviously fuel pickup needs to also be near the bottom, but not beside the return, as the return is warm, and you want cool fuel at the pickup.
Do you have a good service manual yet? I like the later version that is in two separate white 3 ring binders.
Grigg