Here is a thought. When I built my bed, I used a product from Total Boat that is a epoxy wood sealer. You mix the two parts together and apply it to the wood. It is absorbed by the wood fibers and seals the wood. It’s flexible so it won’t crack. It is not UV resistant so you will need to topcoat it with a UV protector. You will also need to stain it first if you go that route. The product has the consistency of water so you could probably use it to fill the worm holes.
You could probably sand your wood down exposing the grain, stain if you want then apply the epoxy and topcoat.
I coated my oak bed with it and it sucked it up. It sands easy. Filled up the end grain really well.

Mike


1940 Chevy 1/2 Ton presently... Almost done
1940 Chevy Business Coupe... In pieces