At Delta it was company procedures, which are "blessed" by the FAA as a part of ops specs.
An issue arose one day when a FAA ASI was going to give a check to the crew. He had a beard and the Captain denied him access to the cockpit, citing Delta's procedures. A semi-major brouhaha followed.
The FAA wanted to take certificate action against the Captain for violating the rule that allows ASI's unrestricted access to the cockpit. The company stood by the Captain as he was following Delta procedures which clearly prohibited beards by cockpit crewmembers, of which an FAA ASI was considered one.
Eventually the Delta procedures were changed to exempt FAA folks with cockpit authorizations from the restriction against facial hair, specifically beards.
As I remember the restriction was as some have already stated and was two fold. One the seal of the oxygen mask, the other the presence of substances that were not compatible with oxygen.
On a personal note, I kind of viewed facial hair as intentionally growing and cultivating something around one's mouth that grew wild around one's ass. Never did make much sense. But then, that's just a personal feeling from a clean shaven one...albeit with long hair as you can see from my profile pix! But hey! I didn't have to put an oxygen mask over my head!