The guy who wrote it is afraid of computers. We all know folks like that - reasonably intelligent people who just don't get them. They're the ones who never figured out all the menus in Windows programs started with File, Edit.
But yeah, risk assessment tools are inherently subjective. In simplest terms, the risk of a night flight by a person who has not flown at night in 2 years is different for the pilot who has 500 hours of night experience than it is for the pilot whose only night experience was to meet certificate requirements.
The goal to me is more about getting you to think about it and, to the extent scoring is helpful, to think about how to score for yourself or your company.
This is a huge problem - the lack of objective risk assessment programs means that there is no actual usable data on how effective or how are useful risk management techniques and training are in preventing aircraft accidents. If it's only purpose is to "get you to think about the risks you're about to take" then it is more likely that we are failing in the training pipeline somewhere else because pilots don't know how to think about "what is dangerous/risky" without the help of a chart.
I am a huge proponent of risk management tools and techniques, I really like scientific tools. That said, it's been my experience in the field that the tools that are available are not scientific, nor do they effectively manage risk. Consider this, we have terms like, "probable, occasional, remote, improbable" that are inadequately defined in the risk management handbook:
The following are guidelines for making assignments.
• Probable—an event will occur several times.
• Occasional—an event will probably occur sometime.
• Remote—an event is unlikely to occur, but is possible.
• Improbable—an event is highly unlikely to occur.
"An event will occur several times?" What? Per flight, per hour, per day? Per minute? Per second? What is the actual probability threshold? Is it "the even has a 75% probability of occurring on each revenue flight?" Do we know? Is this even knowable? What does a "high score" on a risk assessment really fundamentally mean?
If we score up a hypothetical freight flying flight that I used to do nightly here's what I get using the sample FRAT on 4-3 of the Risk Management handbook.
Sleep - I did not sleep well (back side of the clock) +2
How do I feel - Feel a bit off (it's 3:30am, and I'm about to go fly of couse I feel crummy). +2
Weather at Termination - IMC Conditions +4
How is the day going? Great day (night really) +0
Is the flight - Night? Yup. +3
Planning: Rush to get off the ground (always flying UPS packages, gotta start moving so the center can show you blocked out!) +3
Used computer program for all planning - iPad ftw +3
Did you verify weight and balance - theoretically, they told you what the weights were, that doesn't mean they're accurate. If the tailstand isn't touching you're probably OK +0
Did you evaluate performance? - not specifically for this turn, but it's a cold winter day, and I do this flight 8 times a week on 10,000' runways, and I know I need 3800' to make accelerate stop...so kinda +0
Do you brief your passengers on the ground and in flight. N/A +0
I was at 17 basically EVERY night. If I have a cold, or if I am having, "one of those days" suddenly I'm in the "area of concern" - this FRAT isn't useful to me for that mission, and doesn't tell me much of anything.
Now let's say I score a recent flight that resulted in a fatal accident with this thing. There was a bonanza that crashed in Ogden, UT in good day, VMC weather, shortly after departure.
If that guy was having a "good day" and did everything right before takeoff, he would have scored a 2. Even if he had a cold and didn't sleep, he would have scored an 8. Granted, this is kind of an exercise in cherry-picking, because I could pick any accident on a good weather day and screw with the data, but if we are going to have tools, they need to be tools that use real, quantifiable metrics.
In mathematical modelling, in order for a model to be a valid model, the model has to be testable on real world data, and should fit the data, or at least be able to have some predictive quality to it. I'm not sure that numerical flight risk assessments do that.
Even the CFIT Risk Assessment in that book isn't all that great. Where do these numbers come from? I've tried to find out, I find a document from the FSF, but beyond that, nothing. The Flight Safety Foundation says, "studies" but won't give me access to them. In short, there's little that leads me to believe that they weren't just made up by someone because they seemed "risky."
Why does "No ATC Service" score at -30 - the same score as "Limited lighting system?" Why does an NDB approach in Kansas where there's literally nothing to run into score the same as the NDB in Dutch Harbor? This begs a further question - do the minimums of the approach as defined by TERPS provide equal levels of risk at all airports (something tells me the answer is "no").
Regardless - I love this stuff, but it needs a serious overhaul, and the numbers you get from these risk assessment charts are by and large meaningless. They don't "quantify" anything. We need to do some serious model building to see what exactly a "risky flight" actually looks like - as it stands right now, what we're doing is groping in the dark.
