My first instinct was to say you need to give that fence a gap to allow dirt to blow out and wet to dry. Looked at your pics. and I have to stick with my gut; give it a 1/4 to 3/8 inch gap. Over time, I think you will be saving yourself work, trapped moisture and dirt will take it toll.
