Use a guide-coat. Once that's all coming off at the same time, they are ready!
Here's my rule of thumb: If you have spots that don't get touched at all when sanding with a long board, THAT's where you need a little more filler. If the board kinda-sorta touches most spots, then go on to Epoxy and your high build primer.
