ATC gives you that altimeter setting because that is what they are using for radar purposes for that particular area. Pressure can change greatly over a short distance, so your altimeter could read 100 feet off from what ATC sees you at, and in busy airspace areas like LA, you could bust Bravo if you're not careful.
This is how I always handle handoffs. Tune in the new frequency, and make the initial call. I usually say "SoCal Approach (or whoever it is), Cessna ***** with you, 4,500". They'll usually respond with "Roger, altimeter ****". At that point, acknowledge the pressure setting, and adjust your Kollsmann window.