If your concern is really the risk of damage these people are doing to themselves, then the intention behind the sport is irrelevant, only the damage it actually does.
Surprisingly hard to find stats directly comparing various sports for head injuries, but in this Japanese research judo, rugby, baseball and swimming all had higher deaths and severe disability than boxing in school aged children over a 13 year period, whereas this research finds ice hockey has the highest rate of concussion in adult male sports, Taekwondo in adult female sports.
If you want to ban boxing you should ban swimming, rugby and baseball to be consistent, or offer additional reasons it should be banned but not the others.
That's your opinion. You need a better reason than you think it's childish in order to stop consenting adults from participating in an activity - otherwise i'll have to stop playing dungeons and dragons. And more generally, why do you get to dictate how much risk another adult should take with their health?