The shaft on the door handles is steel, as is the latch mechanism. So if you're handy with a welder, you could build up the worn spots and file them down to the right profile. But unless there's a ton of play, I don't think I'd bother. Or you might be able to put some shim stock in along side the shaft. But getting that to stay in place might be tricky.
Good work. Perseverance pays off.

Now might be a good time to modify that latch to use the key lock.
[on edit] good work on the pics, too!