Have you removed the shift tower and examined the gears inside the box? You've probably got a "sliding gear" transmission, commonly known as a "crash box" that has no synchronizers. It's necessary to shift one of those boxes fairly slowly, and let the gear speeds match up to prevent grinding. Downshifting is even more challenging, as it involves double-clutching- - - slipping the shifter into neutral, letting the clutch out, revving the engine to match gear speeds, and clutching again before trying for the lower gear position. Experienced crash box drivers don't bother with the clutch- - - -they just eased up on the throttle, to "float" the gears, slip the shifter into neutral, rev the engine to match speeds, and engage the lower gear, all without clutching. It takes a lot of practice, and could be hazardous to a scarce transmission's health!
Jerry