I believe Patterson would win this. Patterson gets a bad reputation for his two fights with Liston. The same way Spinks gets one for his one loss against Tyson. Floyd Patterson was trained by the late Cus D'Amato and used his blistering handspeed and combinations to become, at the time, the youngest Heavyweight Champion in the world and the first man to ever regain the heavyweight title after losing it. Floyd had back issues starting around 1964-1965 and prior to that he lost to Maxim, Johansson (Which he avenged) and Liston. A prime Patterson takes this fight but will most likely have to climb off the canvas to do so.
I like this match. Both were highly skilled and on the smaller side of heavyweight. Jimmy Bivins had one of the best win lists of all time but was also a bit less consistent and also spent a lot of time at light heavy. I’m leaning towards Patterson by a decision or late stoppage but Jimmy could give him problems