If you're accused of being drunk, having been drinking, high, etc, even if it's an off-handed "joke", you don't really have a choice but to delay/cancel the flight in order to be tested. Imagine a comment like this being ignored, you continue with your flight and, parish the thought, you're plane is hit by a stray bolt of lightning that is born of clear air. Who knows? Or maybe after you land and start to taxi, a baggage guy runs into your plane as you taxi in.
NOW you've got big problems!!! Drunk or not, SOMEONE will recall the comment having been made and will talk about it, how they heard you were drunk and how just because you don't test drunk now doesn't mean you didn't sober up in flight........
Anyway, nothing good can come from idiots making these comments. May as well wave real big and scream "Hi Jack!" just to see the response.....
You can't fix stupid!!