I have done this before, twice. Bay City, MI to Meigs field and back on two separate trips. We went straight across the lake both times at 8,000', we had no floatation gear, and after we got back to Bay City on one of these trips we found out that one of the cylinder heads was cracked.
Oops.
I wouldn't do it again, personally. Your chances for survival are already lower going into the water than onto land, and as the water gets colder, if you survive the ditching, you're unlikely to stay alive much longer, depending on how cold the lake is.
There are going to be a ton of tough guys, on the internet and at the airport, that tell you what to do. What you need to consider is the amount of risk you're willing to take with you and/or your family. When you'll be saving a negligible amount of time by hugging the lake until you get to Michigan before you go direct, it is my personal opinion that if I were to do this trip it would not be worth the potential risk of being outside of gliding distance of land.
But what you need to remember is that whether things go right or wrong with this trip, the person that's going to be strapped into the pointy end is you and/or your family, not the people that tell you what to do, and you need to make the right decision for you. Maybe that's flying across the lake. Make it's taking the extra 20 minutes to go around the bottom half. Who knows, but don't let people influence you by saying things like "You're going to die someday," or, "Don't be a wussy," or, "It's no more dangerous than X." It's your ass, and thus, it's your level of risk.