Turned out that big one *was* the set screw. So I put alot of penetrating oil on the shaft and after it sat a while I got it to pop by carefully using my crowbar. I think the handle must've been pounded on by the PO. My guess is that since the head of the screw is a bit distorted, it might not have been tightened well enough to hold the handle securely. It'll probably keep falling off now like the window crank does!

Thanks for the advice... I just wanted to hear some before I started any brute force stuff.

John.