Hmmm... sounds like it only happens when your clutch disc speed is not matched with your flywheel speed? Not sure why it only happens on downshift or neutral.
Could be pilot bushing as well?
Out of curiosity, did you do your own T5 swap? Did you use a trans spacer plate and end up shortening your front bearing retainer as well? When I did mine, at first I did not shorten the tube, but the clearance was so small with the clutch disengaged that I took it all apart and took 1/4" off the tube to avoid any possibility of rubbing. (It doesn't sound like that is your problem though....)
Last edited by Norcal Dave; 05/04/2017 2:38 PM.