🦈💜
Well-Known Member
And it is something that's equally crippling, and equally prevalent, in private industry. Worse, in many sectors, than you could possibly believe. Infrastructure is a cost center, and as a general rule it's nearly impossible to monetize it. The effects on the bottom line from not investing in infrastructure are generally not considered, because they're part of a long-term strategy and corporate focus is almost always quarterly.
Infrastructure costs money. Safety-critical infrastructure costs A LOT OF MONEY. Even tier IV datacenters running 2N+1 on critical systems, outages still happen... and for a datacenter, infrastructure is quite literally the entire business... and the pressure is relatively high to economize anywhere they can.
Let me give you a concrete example—Security systems, while present, are typically OTS and staffed by people who are inadequately trained on their operation. While sharing the same database, they may have different frontend software and procedures at various DC buildings, leading to different backend data for customers and entities who are authorized access depending on where the customer record was created. Many of them hire NOC, remote hands, and security personnel at the lowest rate; I once social-engineered my way into a datacenter that, at its front door, had iris scanners, thumbprint readers, keycard readers (with PIN auth) within man-traps, and a rigorous front-desk check-in procedure, simply by claiming the cart just inside the door had my hardware on it after a shift change. That datacenter hosted, at the time, such companies as Microsoft (though what datacenter doesn't...), Amazon (ditto), eBay, PayPal, and so on. ("Interconnects considered useful.")
While that datacenter had palmprint readers on the cages, it also had a standard raised floor with no underfloor partitioning, monitoring, or intrusion detection. Simply pulling a tile in a four-post rack with hot-side air blocks gave physical access to any other cage. Had I been a bad actor, I could have compromised the entire system with little effort. (It is considered impossible to secure a system to which someone has physical access.)
The man traps, iris scanners, palmprint readers, keycards, etc, all do represent elements of defense, but the way they're designed and implemented is effectively to provide cheap bling to sell management on the concept. Now, granted, when it's 3am and I show up at a datacenter in pajama bottoms to go balls-deep into a failing storage system, I don't want, as the customer, to spend five minutes waiting for them to cross-check everything—which is why I bypassed it, in this case—but that's one reason that running things so as not to piss off your customers can, when taken too far, be a catastrophic failure.
Computer hardware does not typically run slower over time, nor does software typically develop emergent behavior. Something that was built ten years ago, which adequately serves its function and purpose, does not necessarily need to be updated or upgraded just because it's old. This is a fallacy that has cost money that's very likely beyond measure over the course of technological advancement.
People like shiny things. They like having the latest, greatest, gadgety-est things, and they're often willing to take the risk of infant mortality, emergent behavior at scale, and other badness. In infrastructure, that risk—as with any risk—must be justified. It's not enough to upgrade something just because there's a new version out. (Sidebar—I blame Microsoft for the general attitude of "gotta patch! gotta patch! New update! Updated drivers! New patches! Gotta stay up to date!" ... in reality, it's ridiculous to do that for the reasons mentioned above.)
ATC is not Windows. It is not something that should be run for profit, directly or indirectly. It is a safety-critical component of the national airspace system, staffed by consummate professionals doing an incredible job on a daily basis, built with safety as the number-one priority, and bling far behind.
As above, safe, orderly, expedient. In that order.
-Fox
