There are springs and balls the engage the notches you can see on the shift fork rods, so without the leverage that the shifter gives you, may be hard to move. My guess is you're just not using enough force. They're supposed to be hard to move so that they stay in the gear selected.
