After reading the advice on VAF I opted against my original plan of using a hole saw to make the hole larger. Instead I found a 2inch drum sander to remove material slowly and maintain the curve(hence the large diameter drum).
I placed the flange on the skin in the proper orientation, then using a fine point sharpie I marked where the two met. This left me with a nicely marked circle exactly where it needed to be. I removed the flange and then used the drum sander, chucked in my electric drill, to remove the material to the line. I then test fit the cap and found it a little to snug. I then removed the line and now it fits well.