I have a solution to this. We could have a mandatory test that determines if you are fit to have direct influence. It would take a while to make this test in such a way that it can’t be cheated on, but I feel like this is the best way to solve this issue.
Sure, there will be whining about social rights, but given time, this could become accepted. (The only consequence of this test would be being locked into a “spectator room” if you will where they cannot say anything, but will still be able to watch what’s happening.)
Another solution would be to allow people to vote to suspend one’s permission to partake, and enough suspensions will result in a permanent block. If someone gets enough votes to be suspended, a third party would investigate the person in question and decide if they should be.
This would deter excessive aggression and trolls as well.