Another thing to consider, Inoue doesn't fight like a guy packing glass. Usually guys that Crumble from the lightest touch are defensive minded. He stays in the pocket to exchange far too much to have a weak chin. He would have been stopped by now.
Mayweather was floored clearly by Judah yet inept/corrupt refereeing failed to call it. Also badly wobbled by geriatric Mosley and put into dreamland by Maidana in fight two (lucky for Floyd that right landed right on the bell to end the round because his legs were jelly).
He fights like a guy with full confidence in both his technical and his physical abilities. He's not superhuman in any category, but those guys usually have massive weaknesses in some other area that gets found out eventually. I don't see any such weakness in Inoue. He can box at range, brawl in close, counterpunch, fight when hurt, fight when tired and he never seems to lose his composure or his technical perfection. The only question I really have to ask him is how he comes back from a defeat, but if he keeps fighting the way he does that will likely never be answered.