hevyft,
As others have indicated above, your starting system has one or more things that are marginal. When the engine is hot, starting the engine sometimes requires more "oomph" and your system is lacking in one or more components.
This topic/question/problem comes up a few times every week.
Here is a post that starts with inexpensive (or absolutely necessary) remedies first and expensive remedies last. Just remember that any of the remedies are not wasted effort/money.
Another piece of advice that has been recently posted is that the battery cables can have internal; corrosion that is not very easy to observe. Correct-sized 6v battery cables, ground cables, and clean connection at cables/grounds are probably the biggest culprits contributing to difficult starting (with a cold engine or hot engine).
If you end up having to deal with the starter, some repair parts are available from some reproduction parts vendors; and, NAPA sells a reasonably-priced replacement starter.
