Hold on a second. Take a step back and look at what you just put there. Ballons, remove it from the mix. Not exactly a whole bunch of them floating around now is there. And it's not going to go out of control if the pilot dies. Gliders, same, almost. Not exactly a whole bunch of them flying around. Sport pilots, relatively new to the mix, and not enough data out there to support the argument.
I personally think it's a crazy idea to get rid of the 3rd calss. Personally, I think our medical testing requirments need to be more in line with what you have to go through to get an ICAO 1st class. Not just "Sign here, here, here, can you see the open end of the circle, lets listen to your heart. Here is your certificate."
3rd class is pretty much sticking a mirrior under your nose, and seeing if you can fog it. It's a joke. I've known more than a few who have medical issues that would be grounding that still fly, WITH a 3rd class medical, because they don't report. Remember, regulations are written in blood. There is a reason why we have to have a valid medical to fly.