It might make a huge difference. Frazier had his medical problems going into FOTC but it was coming out of FOTC where they really intensified. Ali stretched him. I'm not sure many fighters could stretch Joe Frazier, even Foreman wouldn't actually stretch him, just KO him. It might add years to his near-prime and make for a much more impressive career. Ali clearly hurt Joe badly over the course of three fights.
This is a good point that fight took years off both fighters lives. But with Foreman, Lyle and Shavers Frazier is going to take severe punishment any way. Fraziers health was only one of the reasons for his early retirement. After losing to Foreman for the NABF his career had nowhere to go he was not getting a title shot anyime soon without some big wins that he was going to have to go through hell to get. Ex champions in that spot often retire. While Fraziers health made this decision easier he might have made it anyway. Elite HWs with financial security and no path to the title will often quit. Just look at Louis's reign. His number 1 contenders kept leaving the picture and most were younger then Joe. One of the upsides to the 4 belt era is it facilitates these sort of matchups. Otherwise you'd have a lot of guys taking their ball and going home.