He won't fight Brook because he knows that if/when he loses to Brook his fallback fight is no longer there for him. It's why he can continue to take fights like Canelo etc, knowing his biggest fight financially won't really be hindered by it.