I personally thought it was Demarcus Corley . Mayweathers first fight at 140 , the scores cards dont show it , but every round was hard fought , and Mayweather was forced to fight every round.
Corley got the **** beaten out of him. Castillo, Chavez, Augustus, and Hatton all gave him tougher fights.
IMO Castillo in the first fight and Oscar,Mayweather was hit more in those two fights than any of his other fights.
Castillo seems the easy answer, though I don't think he was robbed in that fight as many others do. I can't think of a reason to pick anyone else.
I think De La Hoya was actually the tougher fight, I think Mayweather improved since the close fight with Castillo.