Burley was among a group of feared and avoided black fighters in the 1940s referred to as "The Black Murderers' Row," a term coined by writer Budd Schulberg. The group included Eddie Booker, Jack Chase, Cocoa Kid, Bert Lytell, Lloyd Marshall, Aaron "Tiger" Wade, and Holman Williams. George Gainford, manager of Sugar Ray Robinson, admitted that he "bypassed" Burley "because his style was such he would have counteracted Robinson's." He asked Ray Arcel, "Can't you get me someone besides Burley?" Johnny Ray, manager of Billy Conn, shouted at matchmaker Art Rooney, who proposed Burley as an opponent, "No! No! No! I don't want Burley. You can have him for Christmas, for New Year's, or your Aunt Tillie's birthday. But never mention his name again." Fritzie Zivic, who lost two out of three fights to Burley, had his manager, Luke Carney, take over Burley's contract to ensure he would never have to face him again.
I'm pretty sure Robinson said "I'm to pretty to fight Charley Burley" and also didn't want to fight Cocoa Kid. Charley was high risk low reward.
