If I am going to be stuck in an airplane, and airport, for hours at a time, I'd rather be comfortable. I don't particularly like the idea of wearing shorts, and a wife beater, and looking like you just crawled out of bed, like some people do, but then again, I wouldn't go anywhere in public, like that. I don't think a t-shirt, pair of jeans, and tennis shoes is a bad thing.
Since most of my flying is non-rev, I typically always wear business casual, just because I think it is more appropriate when traveling as an employee. But if I pay for my ticket, I want to wear what I want to wear; and comfort is going to be one of my biggest concerns; albeit, I'm still not going to look like a slob.