Don't over look the rockers, lifters, pushrods, valve springs, and fuel pump. Just hearing the knock at or around #4 doesn't mean its in the bottom end.
Side to side shouldn't knock unless the crank is moving around alot. Check the crank thrust bearing, see if you can pry the whole crank forwards or backwards in the block. Also check the flywheel bolts very close. They only have to be a little loose to make a lot of noisy. Any chance the lock washer pushed the rod nut farther out on the bolt letting it hit somewhere? Pull the plugs and spin the motor with the starter, give it 12volts so it really goes, you might hear or see something.

At this point, you only have a few choices, pull it out for a tear down, or tear it apart as much as you can till you find it. Eather way, you are looking at more gaskets and some time.

Joe