I changed mine also, but you don't have to if you have a new license issued from the country you are flying in and don't use the FAA cert for any flying.
I don't have an AIM in front of me but aren't you suppose to notify the Feds of any change of address within 30 days? I remember some FSDO's saying that by not doing so will invalidate your privileges. I do know many pilots that don't change their addy's but it's necessary to make an informed choice. Double check.