How about a closeup of the inside of the switch (although you said it was new, and the contact button on the starter case looks clean), and one of the commutator. That would help find if there's an issue there.
Jerry's comment about making sure the ground path is good (no dirty high resistance connections) is appropriate. You need to identify anything either on hot or ground side of the circuit that would drip the voltage.
Also, be sure the battery has a good charge. A low battery would explain a slow starter.
