Barring ridiculous comebacks like Jack Johnson's, perhaps Ezzard Charles had the least left when the end is compared to his prime.
That's hard, since not even the people in this forum is going to know every fighter. I would have to say that of the elite, Ray Leonard does come to mind. He was once very fast. I believe one of the factors that led to bad showings later in his career was his inactivity. He never learned to fight as an aging fighter. He just jumped in, head first, just like he was the same Ray as before. He never learned to compensate for he lack of speed, nor declining reflexes. Fighters that did well at advanced ages also fought a lot at their advanced age. Foreman took his time. Larry Holmes didn't just fight Mercer staight out of retirement. Mike McCallum remained fairly active until he retired. I don't know of James Toney taking years off from the sport. Walcott and Moore were active until they retired.
Activity does have a lot to do with it; the shades of erosion are harder to pick up when a fighter is active. In Leonard's case, then, the fall seemed sudden and catastrophic.