What was the process that you used, did you sand the complete pan with course paper before adding filler. Then sand it all down with progressively finner grits. I always spray my primer after sanding with 220 and spay 2 heavy coats. When this is dry you can follow with a very light mist coat of black. Block sand with 320. This will show any high or low spots. Then you can either fill with spot putty or more primer depending on condition that you find.
As a side note.....body filler puts up with the temperature of the sun beating down on a black paint job. That creates higher temps than your oil. If your prep work is good you won't have a problem.
