Like others have said, Warning Areas are potentially very dangerous for non-participants. I wouldn't worry a whole lot about the live ordnance part (at least from an aviation standpoint), as live shoots are very controlled, with normally various range spotter aircraft and boats/ships clearing the area for any potential unlucky bystanders........that said, naval artillery still does go down in some of these areas, and I have no idea what they are working with safety wise.
Anyway, the big risk would just be winding up in the middle of a 10-20 jet exercise, with everyone flying at tactical airspeeds, potentially very focused on the mission at hand, and just not having a whole lot of SA to non-participating airplanes. Generally GK or Beaver or whoever will come up on Guard (we normally are up an area common freq and not directly in comms with the controlling agency) if someone comes through, but that isn't always enough. There are plenty of times when I will be inverted (or some other unusual attitude), supersonic, trying to work the radar, maintain my formation, while communicating with AIC, and employing weapon systems. About the last thing I have time for is searching the sky for some random dude trolling through the area who is uncalled by anyone. They aren't "Restricted" areas, because the FAA has no authority to restrict airspace outside US airspace, but they have every type of activity of a restricted area going on.
Get in contact with the controlling agency, and coordinate transit if you need to. Warning areas aren't always hot, and they will often put a temporary cap on our altitude to allow for someone to transit over the top. That gives everyone a warm fuzzy, and will prevent a dangerous near or actual midair. At the very least, it will cue the controlling agency to give us a heads up so we can avoid you. We're all aviators, and I have no more of a desire to die in a fiery midair than any civilian pilot out there. I also have no desire to make you fly 300 miles out of your way just so that I can have the sky to myself. There are very easy ways to deconflict that make everyone happy, so long as everyone is doing the right thing and communicating with one another (or again, the controlling agency).