I see absolutely no reason why your plan wouldn't work. You'd need to be sure that enough screws were used to handle the shear that might occur. I think I'd also use some bearing mount loctite when screwing the washers in place, just as a belt-and-suspenders kind of thing.
