You need good connections at both power and ground.

Since you can push the switch and hear the starter spin, you have proper connections, but corrosion at one or more increases resistance, which makes the starter not turn when the starter gear engages the flywheel.