I poured over the plans... I found that I riveted the aileron hinges with the countersink on the wrong side.
Instead of drilling everything out and possibly causing an issue. I mounted the hinge points and checked the clearance of the shop heads for the ailerons. I drilled one rivet out, and replaced it with a double countersink, flush on both sides. I have not used that technique since the airlines but it worked great and I am pleased with the results.
So long story short... I replaced one rivet... instead of a dozen. I also noticed that in the kit I had (2) right side aileron hinge points A-407-R, although it was labeled A-407-L... no problem, called Van's and they are shipping me the correct part.