This really isn't true in a cut-and-dry sense. For instance, Rogers Mtagwa is a hell of a lot harder to knock out than Amir Khan. Khan, however is still a B+ (maybe flirting with A-) fighter, and Mtagwa is a C, maybe C+ at best. Durability and elite status certainly have some overlap but are not directly and proportionately interdependent properties.
Look at the opposition (ignore KOs) and Pac has not fought tougher opposition. He has had tough opposition, but not tougher. His most recent opposition has been Morales, Barrera, Marquez, DLH, and Hatton. Of those, the only prime one was Marquez and he gave Pacquiao hell (BTW look at how Marquez did against a WW). Morales was close to his prime, and beat Pac in their first matchup, and Hatton was coming off of a knock out loss. Among others, Cotto has fought Quintana, Judah, Mosley, Margarito, and Cottey. His loss to Margo is questionable, but still a loss. His performance with Clottey was below par, but he had one eye. Also lets not forget that he handed Clottey the first knock down of his career. People seem to focus on Pac's destruction of DLH and Hatton, and think he's invincible. They forget he was rocked by Morales, Marquez, and other small fighters. If he was rocked by some of these small guys, do you really think a big guy like Cotto won't be able to rock him? Before you say Pac's too fast and Cotto won't land, don't under estimate Cotto's timing. If he landed on Judah and Mosley, I'm sure he'll land on Pac. Also, Cotto fought his opposition at their natural weights, he didn't try to drain them before getting in the ring with them (in fact, he was drained for his last few jr ww fights).