You mentioned there was someone with three. Keith Thurman has been away for two and a half years. If he doesn't fight again Pacquiao will have retired three. (Thurman, Bradley, De La Hoya) Also, I am unsure if Lucas Matthysse was ever actually a champ. If he was, Manny retired him as well.
Matthysse was indeed a world champion - so if both he, Bradley and Thurman (and DLH!) never fight again, Manny may actually end up having retired 4 world champions! But whether or not that turns out to be the case, he would not be the first to retire 3 world champions. There were some before him.