I figured out I had to find a better way to keep things in position while drilling. I match-drilled a few of the holes in the skin and flap brace first. The I used oops rivets to hold the brace and skin in position. The smaller heads on the oops rivet does not interfere too much with the hinge. I positioned the flap/hinge again, aligning and trailing edge and assured the 1/4 distance with some AN bolts. I could now drill half of the holes in the hinge with #41. I disassembled, deburred and could now use clecoes in the newly drilled holes. I then drilled the remaining holes, previously occupied with oops rivets with #41. I went over all the holes with #40 reamer. Drilling is now done.