"Gateway Ground, N12345, unable taxi at this time."
If he presses the issue, say "I did not feel it was safe to taxi at this time due to traffic that was approaching, I am solely responsible for the safe operation of this aircraft, if you feel differently, please mark the tapes so we can discuss the meaning of FAR 91.3 at a later time."
That should get his attention, and if he says to call the tower, I would say, "Gladly, and when I'm calling, I'd like to speak with your supervisor."
You have every right to stop if you want, to turn around, and also to delay. You have to notify ATC as soon as practicable, if it was not yet possible to notify them before he started getting pissed, that's a lick on him. If safety is in question, then you can do exactly as you please.
Stick to your guns.
-pat