This morning I putzed around with the sub panel idea. Not sure how this is going to end up but the idea is to mount the Switches to this panel then mount it to the back of the panel. This way the switch nuts would be in between the sub and the panel, then Countersink screw to hold that to the panel.
Got a good fit behind the panel
Matched up the holes in the panel with this sub panel