My private was by far the most difficult for me. There were times during that when I despaired of ever completing FA110, but I got through it ... with an embarrassingly large number of hours for a private pilot, but that's another story. After that, I really had it pretty easy ... my instrument, commercials, CFIs, and even the King Air training was pretty straightforward to me. I think it has a lot to do with what your particular aptitudes are (some people take to IFR easier than VFR flying, etc.) and how well your style and your instructor's mesh. Can't over-emphasize the importance of selecting a good instructor, and not being afraid to make a change if you're not happy!
Russ