I suspect the simplest reason is there are many more regional pilots than there are openings at the majors. The competition greatly exceeds the number of available jobs at the majors and always will. Not everyone can or will be a success story
I honestly don't believe people who claim to not want to go to the majors because they "can't afford a pay cut" or " weekends off", I always figure those people just can't get a call and don't want to admit it. I can't imagine anyone voluntarily forgoing an opportunity to escape the regionals.
At the same time there plenty of pilots who think they're entitled to a job at mainline and so don't put in the effort- all the outrage about the requirement for a college degree (which as far as I can tell has existed for decades at most carriers), etc. These types often have
bizarre conspiracy theories about how mainline "only hires gays, women and minorities"- I remember a jumpseater I once had who had been at ASA for 20+ years (although he'd recently gotten hired at AA), and he claimed that Delta hires a lot of (of course) "female minority" pilots with bad records if they sign a promise never to upgrade.