And many 6'3 220lbs men don't think they need to fight, they've never been bullied etc so they never felt the need to learn how to defend themselves.
You are overestimating the number of people who take up boxing because they are bullied. More often the bullies are interested in it.
Men that big are used to being the big man on campus, and when they get hit back, they crumble mentally.