Elmer Ray's record going into the first Walcott fight was admirable. A long streak of some 40+ fights and a lot of knockouts. His record has been scrutinized for having some padding, but he was definitely an avoided man during that period.
Quite a few articles of the time claim Ray to be avoided. I believe Baski most notably turned down an offer to meet him. He did demolish Savlod, who was a sturdy contender.