When you say rough spot, do you mean the primer was still soft?
What kind of primer are you using?