He knew it was coming though and handled himself accordingly by not engaging in any pre-fight talk & Fury was nice to him because there is no need to motivation for him to put Whyte down. He's already done that.
Sulaiman protected him from Whyte years ago, if he didn't Whyte would have went to Brooklyn and knocked Wilder out cold.