If your lethargic from the weight it's possible that you will slow down as your tank empties because you're so unable to operate at that weight that it's impossible for you to perform. But that is rarer and rarer. People underestimate how much "rehydration" has come on. It's not even rehydration any more. Mikey Garcia would come off the scales unable to make a fist but the following night he'd be a world class athelte. I'd be astonished if Naoya was faster due to rehydration. Most of all, he wouldn't know yet.
He's definitely bigger than some of the guys he fought at 108-115. Personally, I don't think that diminishes his accomplishments at those weights in any way because Inoue, like almost every other fighter, historically has fought in the lowest possible division he can make effectively. You can't excuse Inoue on the one hand and then accuse other guys on the other hand when they're all doing the exact same thing. That's how boxing has worked for about a century now and why the weight bully concept is just silly fan fiction.
Speed is not really affected with a hard weight cut but you definitely feel like you're fighting 2 opponents: the other guy and yourself.