Point #1: If this guy is completely self insured, I'd never step foot in his plane again. EVER! It's okay if he wants to assume risk for hull damage, but he needs to have liability and medical coverage. Ask him to bring in his insurance papers for you to review, and then insist he lists you on the policy. You'll also need your own insurance because even if you're a named pilot, it's likely not the same as being the owner. You'll need to talk to his agent (and yours) to find out all the details.
While you're at it, you should review the aircraft logbooks. You need to give him a lesson on them anyhow, so you might as well do it now. The last thing you want to do is have your certificates revoked for flying a plane that isn't airworthy. Chances are his comments about you swindling him is simply an indicator of him being a cheapskate, so I'd expect him to treat his airplane in the same manner.
Now, on to the quandary: Communication is the key here. I had a student like this once -- he was always concerned about how long it was going to take him to solo, he reviewed every lesson on the phone with his uncle afterwords, he constantly compared his flight time to everyone else on the planet, and he always felt like he should be farther ahead of everyone else; when in fact, he was a below average pilot. I finally had to sit him down and spell it out to him, and I recommend you do the same. In fairness, my customer came to me with a certain expectation of what it would take to get his pilot's license, but I didn't do a good job of changing those expectations in a timely manner. His expectations were way off, but there's no way he would know that until I explained it to him. The fact that I wasn't meeting his expectations made him seem pushy and impatient to me.
I recommend you sit your student down and outline your requirements. Make sure he understands that although he owns the airplane, he won't be able to go flying in it anytime he wants; he can only fly the plane solo with your permission. (After I found out my first solo student went flying without telling me about it, I always add "must receive my permission for each solo flight" on his solo endorsement along with other limitations for wind, weather, and currency. I would also tell him why you will not permit him to fly from his local grass strip. One of your responsibilities as a CFI is to provide a safe environment for your students to learn and grow as pilots. I do that by placing very protective limits on them in the beginning, but as they get more experience and skill I relax those limits. For example, I start solo students off with a 7 day currency limit -- if they haven't flown in the previous 7 days they must fly dual for one flight before they can solo again. Normally I'll raise this to 10 days about the time they start solo cross-country phase, and up it to 15 days by the time they are in the test prep phase. (My school also has a limit of 30 day or 10 flight hrs between dual flight for students, otherwise I'd have it too.) My goal is to build a box that is smaller than their skills, so they won't get into a corner where they don't have the skill to get out. When their skills increase, I increase the size of the box. When I explain to each pilot why I do this, most of them appreciate it. It's natural for a student to feel a little restricted (he is restricted), and even to complain about it a little (I remember saying, "Being a student sucks!"); but they should all realize these restrictions are for their own good. The bottom line is they don't have enough experience & skill to be a private pilot yet or they would be one.
I would have a talk with your student before you schedule another lesson. If you lay out your expectations and limitations and he isn't completely supportive, I'd part company with him. If you decide to keep him as a student, make sure you place limitations on his solo endorsement. Otherwise, you could solo him and he can legally fly for the next 89 days on your endorsement without so much as a phone call to you. You can't take back an endorsement once you give it, so make sure you place some controls on it.
One last thing, you should also find out why he left his former instructors. If possible you should talk to them and get their side of the story. Finding an instructor who works well with a student is a natural part of learning any skill. Maybe they just didn't get along or whatever legitimate reason, but maybe he couldn't get his way with them and he set off to find someone weaker. If you find out the latter is the case, you'll then know what your student thinks of you.