Because by the time Vitali came back, Byrd had tucked tail and degenerated into a shitty light heavy/cruiserweight.
Many rematches never happen. We were quite lucky to see some historic rematches, e.g. Lewis' fights against McCall and Rahman. McCall could have gone to drug rehab (he should have) and Rahman/King could have won in court and Lewis would never have had those brilliant rematches. It's all a bit of throwing dice for them to happen.
As much as I couldn't stand Byrd's style, and those obnoxious facial expressions he'd make (well beyond the Michael Nunn smirk), I always respected him for fighting guys far larger than him. Byrd is like 6 feet tall, fought at middle in the Olympics, and was carrying a lot of extra weight at 210. I'm just looking through his record and Ike Ibeabuchi had over 35 pounds on him. Ross Purrity had 34. Vitali is about 7 inches taller and had over 30 pounds on him. Wlad had 25+ lbs on him and is half a foot taller. Jameel McCline is half a foot taller and had 56 pounds on him. Golota is 6'4 and had 27 lbs on him. David Tua was in shape and "only" had 18 pounds on Byrd. Byrd hurt him to the body in the 8th round.