Hopkins was P4P number 1 in 2005 prior to the Taylor losses, at 40 years old. If we go by unnoficial, then Archie Moore absolutely must have been P4P number 1 in the mid 50's when he was in his 40's.
Had rankings been around at the time maybe Bob Fitzsimmons was P4P number 1 after beating Ruhlin and Sharkey in 1900. He was nearing 38.