I'll be the first to admit flying isn't for everyone. Nothing wrong with that.
Honestly, my first 20 takeoffs or so were very much a love/hate thing. I'd be driving out to the DZ, hoping the wind would be too strong, or the clouds too low, or some excuse to cancel. Even sitting on the ground, my heart rate would go up and I'd lose my appetite just thinking about flying in a few minutes. But then I'd force myself to make a flight and the feeling of dread would be overwhelmed by the adrenaline, sense of accomplishment, relaxing view while hanging out under the wings at sunset, etc. I couldn't quit.
I think it's something most pilots push through, in one way or another. The more I started packing for myself, trusting the equipment (and seeing how forgiving it actually is), trying different things with the plane, gaining landing skills, etc. the more confident I got. Even now, on that last 1000 feet of descending, right before the runway comes up, I feel my heart rate go up. But it's totally manageable. As soon as I step out on to the ground it's a huge sense of peace. Time slows down and my mind shuts out absolutely everything except what's happening in the moment. The heart rate drops, the fear goes away, and I drive off, knowing that I'll fly again.
Good luck to you if you decide to get back in to it. Let me know how it goes!