I think the paint on the axel mount area of the frame acted like lubricant and allowed the axel to shift position. Cleaned up all the mating surfaces and re-attached the trailer mounts.
Added some rub protectors for the panniers so the bolts won't rub through the panniers.