GypsyPilot
Mohawk Town
Do it.
If it doesn't work out, there's always medicine.
This. I can't tell you how many times I heard "Just buy an airplane and fly it on weekends!" Uh, no? It always seemed to be those pilots who never worked a day outside the industry that said that the most too!
When I left my career (mechanical engineering) it was nothing like medicine. But it was still a lot to leave and was a tough decision. But for me, it was the right one and I don't regret it a bit. As Doug said, I can always go back if I need to.