Sam Langford was a MW and LHW, and he was ducked even by the heavyweight champion of his time. I'd say that puts him among the most avoided.
For a period of time he was but he still fought Vargas, Trinidad, and Mosley twice! Not bad at all, really Oscar was the only one who decided there was better money with other bigger name fighters so wasn't worth the risk/reward and winky isn't a fighter you look good against even if you beat him. Oscar would have outboxed him easily IMO, winky had minimal power and not great speed either, but covered up well and was sneaky. Amazingly enough Vargas even beat him!
Burley is always brought up, highly skilled, but SRR beat fighters just as good. Pryor is another, maybe at 140 he reigns supreme, but not at 147. He'd have some success, maybe similar to Duran, but as athletically gifted he was he was way too reckless. Tommy was at his best 154, perhaps if Pryor brought his power and stamina from 140 to 147, but I'd still favor the Fab 4 to beat Pryor above 140.
Winky struggled always with volume punchers like Soliman and unsurprisingly Williams, prime or not. He was a puncher's nightmare cuz aside from a tight defense he was a strong 154lber so he absorbed big shots well. That's why tho outgunned by stronger punchers, Winky always was willing to trade using his tight guard and southpaw jab to pressure and wear down and basically outmuscle a fighter. Mosley and Tito usually outmuscled or outgunned their opponents. Winky was a tank. Oscar that got robbed in the rematch vs Mosley had a great chance to beat Winky. That Oscar was sharper with faster hands and better stamina.
Definitely Mike McCallum in his prime. I also read that some bad ******* serial ****** called Tony Ayala was avoided.