Thinking about Ron Lyle, he started boxing well into his 20s and became a top rated contender in one of the best heavyweight eras ever. I believe that Wilder didn't start boxing until 20. It would be considered impossible to make it to the NFL or NBA and you didn't pick up the sport until your 20s. Let alone be one of the top players.
Lots of sports involve sprinting, which you peak at around 24-28, jumping also, boxing is very technical and is about thinking steps ahead based on opponents mistakes, the most important things boxing involve are things which stay with you, I’d say 70% is technical, IQ etc and 30% is physical, and even in the physical department, strength and punching power doesn’t decline very much. Things like reflexes and foot speed decline with age, but if you’re able to think steps ahead offensively and defensively, you can still remain very good even with age. One of the main things that declines with age is how fast you can cover ground with your feet, which is why footballers, sprinters etc typically decline faster. Heavyweight boxers are able to go on longer than other weights mainly because they’re slower in terms of punches and less reliant on reflexes.
Most sports leagues have official or non-offical farm teams or minor leagues, whether they are schools, private clubs, universities, or whatever. The top coaches and scouts tend to have their eyes on players and follow their progress long before they ever make it to the top leagues, if they ever do. You can't just decide in your 20s you want to play major sports in the US and go try out. Their tryouts are by invite only. The players on each pro team were usually the very best player on every team they ever played on their entire lives, since they were kids. Then they turn pro, and everyone on their team was the best player their whole lives, and the competition gets a lot harder. You can't just decide later in life you want to do it. Boxing isn't a league. Their aren't owners. So you can get involved any time you like for free. So lots of people who have no business doing it show up in the ring.
Because Boxing is an art and also has a huge, huge mental component. There is no 'perfect' way to do it. Things like sprinting, jumping and even some other sports are pretty rudimentary in comparison. It is entirely possible to be outgunned in every physical facet and still win a bout by using a good game plan. That's not something you'll really see anywhere else, especially in an individual sport. There's also the fact that people focus mainly on higher level boxers who are going to be exceptional in some / all traits anyway. Even someone who is 'only' British level, for example, is massively exceptional compared to the average. This skews peoples perception. For reference, someone like Lyle would absolutely be rated as a 'world level' talent imo. Also when people talk about traits in Boxing they usually focus on reflexes, speed etc like in other sports. But where Boxing is actually highly selective is in terms of toughness - which doesn't really need a young age. A great chin is probably the best attribute you could possibly have in the game. Combine it with technical schooling and you'll have nightmares boxers who stay around (see: Marcos Villisana). The higher in weight you go the more apparent this is. A final thing I'll say is that though the more rational types hate to hear it, is that some people absolutely just have the 'knack' when it comes to fighting. It's not something you can really explain or 'prove', but once you see it you know.
There is such a variety of styles in the sport, each fits every person differently. Some styles don't require explosive/athletic attributes. E.g Beterbiev is 40 years old at the highest level, which is insanity. Artur is athletic and gifted with strength, but his clubbing style isn't required for Mike Tyson esque athleticism Also, it should be noted Heavyweights don't require as much technique as a Flyweight or Middleweight. While it takes incredible skill, at heavyweight, being athletic with decent technique can get you far (in this day in age, probably top 10 lol)