There are well-known and high-quality epoxy primers that allow body filler to be applied over it (as long as appropriate procedures are followed). This is very commonly done during restorations.

What brand of epoxy primer are you using?