Most times you want to turn the nut to where it is snug and all looseness is gone.
Now that don't mean put a big wrench on it and crank till it's tight, but just to the point where it's snug. Then back off that castle nut to where you can insert the cotter pin.
I usually use a small channel lock, that way I can feel the amount of torque I'm putting to it.
If you want to acquire the feel for it, tighten it, back it off,tighten it, back it off, and you will feel the the point at which it gets snug.
A tad on the loose side is better than to tight.
Dan