He was already #1 IMO... Soon as he got out of his contract and signed another one he was ready to fight Khan, that there makes me believe he wasn't ducking Khan but didn't like the situation.
Put it this way, the money he would have made from fighting Khan (1.3 mill at the very least, and extras), would have been better than sitting idle for one fight before eventually facing Casa on an undercard for something I'm pretty sure wasn't that much. And come on, he signed with Bob Arum. A man who keeps his biggest fights in house and doesn't get along with GBP anymore. Did he, Bradley, honestly think a fight with Khan, a GBP fighter, would be that easily available under these new circumstances? He boarded another ship, and it sailed. He was just tryna' save face at that point.