Yes, lotsa ambiguity here. It sounds like he wants you to stay off Alpha on your way back to the ramp, and that’s all he cares about. He may not realize that you cleared Alpha and he wants you to hurry and clear Alpha, that’s what he might be frustrated about. I wouldn’t guess in a case like this.
When I get an ambiguous clearance like this, I try to avoid asking for repeat of the same ambiguous clearance that might elevate everybody’s frustration. I’d probably reply, I’m Lima short of Bravo, requesting Lima all the way to the ramp.