...checklists and human factors...breakaway thread

[tgrayson]

...but what put you at odds with John <Deakin>, the 'blowhard'?

John's views are formed to a large degree by a contrarian attitude, which makes him rather tiresome. In many ways, this means that other people control what he thinks; all "they" have to do is advocate something and he's against it. To me, this makes his judgment unreliable, yet his AvWeb pulpit has made his judgment influential.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

John's views are formed to a large degree by a contrarian attitude, which makes him rather tiresome. In many ways, this means that other people control what he thinks; all "they" have to do is advocate something and he's against it. To me, this makes his judgment unreliable, yet his AvWeb pulpit has made his judgment influential.

This will just have to be another issue we disagree on.

John's articles on training and checklist got him in hot water with the FAA and were considered heresy (contrary) but he was right on with many points. John's willingness to take on issues that may come back to bite him tells me he is not controlled by others.

I assume you know John suffered a stroke a few years back.

 

[tgrayson]

Re: Combinations of RPM & Manifold Pressure

John's willingness to take on issues that may come back to bite him tells me he is not controlled by others.

That's one of the natural results of being a contrarian; in fact, the outrage is probably one of the payoffs. John got into trouble with the checklists article because he shot his mouth off by overstating his case to generate controversy, and he paid the price for it. That's what a blowhard does. I'm sure that a calm, reasoned, moderate article would not have been nearly as popular, but it would have been more accurate and useful. John's article on the checklists generated a response from an FAA inspector that I thought was very insightful and more in line with my own philosophy than John's article, and benefited from far more years of evaluating pilots than John can boast. I posted that response on the forum that John and I were involved in and he admitted that he thought the inspector's response was his favorite. I'll see if I can dig it up and post it here.

I assume you know John suffered a stroke a few years back.

No, I didn't know. Our previous forum (on Compuserve) didn't make the transition to the web very well and it effectively disintegrated. I haven't had any contact with that crowd in 7 or 8 years.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

That's one of the natural results of being a contrarian; in fact, the outrage is probably one of the payoffs. John got into trouble with the checklists article because he shot his mouth off by overstating his case to generate controversy, and he paid the price for it. That's what a blowhard does. I'm sure that a calm, reasoned, moderate article would not have been nearly as popular, but it would have been more accurate and useful. John's article on the checklists generated a response from an FAA inspector that I thought was very insightful and more in line with my own philosophy than John's article, and benefited from far more years of evaluating pilots than John can boast. I posted that response on the forum that John and I were involved in and he admitted that he thought the inspector's response was his favorite. I'll see if I can dig it up and post it here.

John's checklist article was intended to stir discussion and it did. As I remember John argued for the use of 'flows' more than 'do lists' and the title about throw the checklist away cost him his DPE status. I would like to read the inspector's response.

As for evaluating pilots, with almost 40,000hrs I would think regardless of FAA tickets, one would have a pretty good idea of who is real and who is a pretender.

 

[tgrayson]

Re: Combinations of RPM & Manifold Pressure

I would like to read the inspector's response.

Wow, I saved this back in 1998:

===================================


[FONT=Times New Roman, serif]To:[/FONT][FONT=Times New Roman, serif] John Deakin - jdeakin@avweb.com [Read][/FONT]
[FONT=Times New Roman, serif]Fm:[/FONT][FONT=Times New Roman, serif] Fred Harms - Fred_Harms@hq.faa.gov [/FONT]
[FONT=Times New Roman, serif]Dt: 13 Apr 1998 [/FONT]


[FONT=Times New Roman, serif]John [/FONT][FONT=Times New Roman, serif]:[/FONT]

[FONT=Times New Roman, serif]Just as in flight instructing, you never really learn about aviation safety until you try to write about it. [/FONT]

[FONT=Times New Roman, serif]I read your original column and I understood the context in which it was written. In the Practical Test Standards, the FAA advocates that the use of a checklist is dependent on the specific task being performed and recognizes that proper scanning and division of attention may render the use of a checklist as impractical, especially in a single-pilot operation. The proper use therefore, is as a "check", not a "do" list. [/FONT]

[FONT=Times New Roman, serif]The use of memory aids has also been accepted as a good procedure for accomplishing a task. The FAA "On Landings" safety program promotes the use of a GUMP[/FONT][FONT=Times New Roman, serif]’[/FONT][FONT=Times New Roman, serif]s check, but in all cases, when division of attention is not compromised, all actions should be verified by a checklist. [/FONT]

[FONT=Times New Roman, serif]As an FAA inspector, retired Army instructor and flight examiner, I have given many checkrides. It is pretty easy to spot those pilots who rely on a flow method as a sole means of completing cockpit tasks, and I have observed that it only takes a minor distraction to cause a disruption in the flow. It often results in important items being missed. When a flow pattern is used, we are relying on our memory alone. A self critique is a great idea, but it only can happen when we realize that we've missed something. [/FONT]

[FONT=Times New Roman, serif]One of the very finest checkrides I observed was for a young man taking a VFR single-pilot Part 135 evaluation in a Cessna 172. The pilot stuck the checklist in the plastic door molding just above his left shoulder. At each appropriate point, he would reach up with his right hand, pull down the checklist, quickly scan it to verify he had accomplished all the tasks, and return it to it's holding place. He used it for every operation throughout the flight, and he did it so smoothly that he never disrupted his traffic scan or negatively affected his division of attention, even in a high workload situation. It was such a natural part of his flying that the entire operation was seamless. He verified his flow with a checklist to know. When I hear people complain that using a checklist, even in a relatively simple aircraft, is an unnecessary burden, I'm reminded of the superior job this young man did. The other great thing is that he is also a CFI, so I'm sure that his students are learning to be equally capable. [/FONT]

[FONT=Times New Roman, serif]When a 32,000-hour pilot writes, "Throw Away That Stupid Checklist!", not everyone is going to understand the context in which it was presented. [/FONT][FONT=Times New Roman, serif]I understand your desire to be controversial and spark debate. Certainly, attacking one of the first, and most basic things we learn as pilots will accomplish that, but your words may have a more profound affect than you realize. [/FONT][FONT=Times New Roman, serif]Considering this week's headlines - "Transponder Found Off In GA Midair" (AVweb Top News Stories, April 13th), your comment: "If you forget your transponder, the worst that will happen is that ATC will say, "confirm squawking," generally with a little chuckle, because they know what you did - or didn't do" might be the truth as you perceive it, but you may be doing a disservice to a relatively new pilot who accepts your words as "gospel". It might just be that the "worst" that can happen is not a chuckle from ATC. Could a checklist have made a difference? I don't know. We never will. In my job, I've seen many things that you eliminated from your "killer" list that have led to tragedies. [/FONT]

[FONT=Times New Roman, serif]As I said in the beginning of this long winded thesis, you never really learn about aviation safety until you try to write about it. I commend you for stepping up to the plate and giving it a shot. Words are powerful things. They can produce positive or negative results. I hope you will continue to contribute to AVweb and that your words will always produce the positive results I know your looking for. [/FONT]

 

[shdw]

Re: Combinations of RPM & Manifold Pressure

My 02:

I teach:

A) Do it as a to do list: Item one, check...item two, check...and so on, pointing to each item as you move through the list.

B) As a check list: Learn, and practice on the ground heavily, a basic flow check; then verify it with the checklist.

The only time I ever advocate a flow check, without checklist verification, is on an emergency where there is not enough time. An example would be an engine out at 300' after takeoff.

The checklist is there because we are all morons, let's just fess up to the facts (statistics of human retardation) and start use them religiously. :beer:

 

[mshunter]

Re: Combinations of RPM & Manifold Pressure

The checklist is there because we are all morons,

Speak for yourself. I never use checklists, and won't let fat chick on my plane because the props will scrape! :bandit:

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

Wow, I saved this back in 1998:

===================================


[FONT=Times New Roman, serif]To:[/FONT][FONT=Times New Roman, serif] John Deakin - jdeakin@avweb.com [Read][/FONT]
[FONT=Times New Roman, serif][/FONT]
[FONT=Times New Roman, serif]Dt: 13 Apr 1998 [/FONT]


[FONT=Times New Roman, serif]John [/FONT][FONT=Times New Roman, serif]:[/FONT]

[FONT=Times New Roman, serif]Just as in flight instructing, you never really learn about aviation safety until you try to write about it. [/FONT]

[FONT=Times New Roman, serif]I read your original column and I understood the context in which it was written. [/FONT][FONT=Times New Roman, serif]. [/FONT]

I went back and read John's piece. Yes it is controversial and the response from the FAA is what one would expect. Don't turn on the transponder and you MAY die.

First, I have spent many many hours studying checklist design including reading the NASA studies including Weiner's early paper. I assume you have read it but it drills down to font and font size as well as noting how poor many checklists are. I have seen checklists which are too long and ones too short. Others poorly designed with seemingly little to no flow to them. Others have useless responses such as 'as required/as desired'. And I have watched countless sims where guys upgrading/doing recurrent/transitioning said all the right things but failed to verify actual switch position or response (failing the flaps is just TOO easy but makes a nice lead in for the stall series). And then there are do-verify and read/do checklists. And in most cases with complex aircraft, many operators have done away with most memory items knowing that crews under stress quite frequently fail to remember items at a critical time so a read/do is best. (our engine fail checklist is ALL read/do and on takeoff, you do NOTHING until the airplane is cleaned up. Any actions other than flying the airplane was a critique item).

John did miss the mark slightly in failing to come back after he got the reader's attention. His ending sentence recommends to use flows and follow up with something mental and written if necessary. In my case, I always vote for the checklist with flows (do-verify). But we know even these are not infallible. What happens when the flier THINKS he has done the flows and thinks he has read the checklist? I saw that happen frequently in the sim with high task loading.

At any rate, John paid a heavy price for his comments. Higher than I thought necessary but then in most heirarchies the rule is often 'for every action there is an unequal and opposite over-reaction." This is necessary to show who's really the boss. Don't know this inspector and have no clue as to whether he is a good guy or not.

Finally, this is not an anti-FAA response. I worked with some really great (note not just good.. GREAT) inspectors. We also had a few who seemed clueless. One I remember said the TRUE measure of a GOOD training program was a bust rate of at least 15-20%. Moron. Former USAF check pilot, active reserve 0-6 and an all round jerk. In fact, now that I think about it...I felt about him sort of like you feel about John.

Have a good weekend.

 

[tgrayson]

Re: Combinations of RPM & Manifold Pressure

studying checklist design including reading the NASA studies including Weiner's early paper. I assume you have read it but it drills down to font and font size as well as noting how poor many checklists are.

I haven't read it, but I'd like to. Do you have a source?

At any rate, John paid a heavy price for his comments. Higher than I thought necessary

Agreed. The FAA appears to contain vindictive elements within it.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

I haven't read it, but I'd like to. Do you have a source?

At your service...

This is one of the first. Degani and Wiener

http://ti.arc.nasa.gov/m/profile/adegani/Cockpit Checklists.pdf

procedures
http://ti.arc.nasa.gov/m/profile/adegani/Design and Operational Aspects.pdf

http://ti.arc.nasa.gov/m/profile/adegani/3Ps.pdf

http://ti.arc.nasa.gov/m/profile/adegani/Human Factors of Flight-Deck Checklists.pdf

http://human-factors.arc.nasa.gov/publications/Burian_EAChecklists_Expect_Realities.pdf

http://human-factors.arc.nasa.gov/publications/Burian_EAChecklist_Design.pdf

A properly designed checklist is truly a work of art and science. One of our guys worked long and hard on our Quick Reference Handbook a while back it it was truly genius. Go into a abnormal/emergency checklist and it took you ALL the way through ALL the necessary procedures to landing. No mixing of emerg/abnormal and normal which I continue to see.

The thing with checklists is how much does one want to learn about how to design and use a good checklist, not only in aviation but anywhere. And as said, a good one is rare and magnificent.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

I forgot to add Dismukes

http://interruptions.net/literature/Dismukes-ISAP01.pdf

and google 'checklist design dismukes' for a good powerpoint. scroll down and you will find two or three powerpoints on concurrent tasks, critical defense against threats, etc. All good stuff.

 

[tgrayson]

Re: Combinations of RPM & Manifold Pressure

At your service...

Thank you! I looked for the NASA stuff years ago when a lot of this was being discussed, but never could locate it. I find Human Factors stuff absolutely fascinating. I've spent a lot of time designing checklists that I give to students, but I didn't really have any valid design philosophy.

 

[USMCmech]

Re: Combinations of RPM & Manifold Pressure

A properly designed checklist is truly a work of art and science.

The checklists provided by the manufacturers in the AFM/POH are worthless and are mostly legal CYA lists.

I like preflight checklists to be read-do, you're not going anywhere so take the time to set up the aircraft the way it needs to be. Normal inflight lists should be mostly flow, with killer items double checked so you can keep you mind on job #1.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

Thank you! I looked for the NASA stuff years ago when a lot of this was being discussed, but never could locate it. I find Human Factors stuff absolutely fascinating. I've spent a lot of time designing checklists that I give to students, but I didn't really have any valid design philosophy.

Had I known about it and had their been a major at the time I was in school, I would have probably majored in hufacts.

I ran into Wiener in EWR back in the 80s. I didn't at that time know who he was. I was reading a book on human factors while waiting on a bunch of crazies who were all riding over to Sweden to tour and write about SAAB. He saw the book, came over and started the conversation. Amazing fellow with a good sense of humor. He accurately noted that with every new technology or gizmo a new error is introduced. As evidence he noted that prior to retractable gear there were no gear up landings.

I've attended a few conferences with Helmreich, Orasanu, Fischer and communicated with Dismukes and Berman. Dismukes book on the limits of Expertise should be mandatory reading. I like Sidney Dekker's stuff as well as the stuff put out by Tony Kern. I sat in on an interview with Dianne Vaughan who wrote about the Challenger disaster. All very interesting and incredibly passionate people about this stuff.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

The checklists provided by the manufacturers in the AFM/POH are worthless and are mostly legal CYA lists.

I like preflight checklists to be read-do, you're not going anywhere so take the time to set up the aircraft the way it needs to be. Normal inflight lists should be mostly flow, with killer items double checked so you can keep you mind on job #1.

Agreed. we have gone far afield of RPM and Manifold pressure. maybe we need to start another thread. ??

Flows with do-verify. Emergency stuff as read/do since the progression of steps is important.

Finally, most of us react properly to the horn/red light. The problem is the slowly evolving problem where the information is ambiguous. I can't remember where I read it but I like the quote which sums it up, "We are constantly moving from ambiguity to ambiguity." We seldom have all the info we need and the other side is when all hell breaks loose, there is a tsunami of info coming in that we have to wade through to find the relevant and useful bits.

 

[tgrayson]

Re: Combinations of RPM & Manifold Pressure

I've attended a few conferences with Helmreich, Orasanu, Fischer and communicated with Dismukes and Berman. Dismukes book on the limits of Expertise should be mandatory reading. I like Sidney Dekker's stuff as well as the stuff put out by Tony Kern. I sat in on an interview with Dianne Vaughan who wrote about the Challenger disaster. All very interesting and incredibly passionate people about this stuff.

I have James Reason's "Human Error", Kern's "Redefining Airmanship", and a somewhat related title of "The Psychology of Everyday Things". Reason's book is somewhat tedious in its presentation, but I think it serves to lay a foundation of the vocabulary of the discipline. He carefully defines words like "error", "mistake", and "slip" as they will be used in the Human Factors literature. I enjoyed Kern's descriptions of various accidents, but I didn't derive much value from his prose between accident descriptions.

For me, the wisdom underlying Human Factors concepts is that it steps away from the macho attitude that the solution to human error is more training or getting rid of those who can't measure up. While those things have a role, there is a limit to how good human beings can be. Human Factors concepts understand and accept this; rather than trying to change the people, they advocate changing the task so that it becomes doable by non-exceptional individuals. That's one reason I don't have much sympathy for those who strongly advocate mandatory spin training for pilots; more training has very diminishing returns. Better to change the task so that the unattainable level of skill isn't necessary.

I see this in my own field, too. People see quite happy with complexity and view the resulting errors as due to people just not trying hard enough. My view is that any process that lends itself towards people making errors needs to be redesigned and this is almost always possible to do, if you accept the need.

I finished reading the first paper you linked to; one thing I noticed is that the list of recommendations at the end didn't seem to have any research associated with them to demonstrate that they were effective in solving the problems they were targeted towards. I can easily imagine such experiments, so I'm sure that a professional with academic training in the field should be able to do so.

I did find the paper useful in define some terminology that is useful when discussing Human Factors issues; one is coupling/decoupling. For instance, when performing instrument training, I encourage students to "decouple" airplane configuration from ATC interactions. That means getting the airplane configured early in the approach, rather than waiting to the FAF. This isn't the most efficient airplane management approach, but, as the article pointed out, while coupling is more efficient, decoupling is more reliable.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

I have James Reason's "Human Error", Kern's "Redefining Airmanship", and a somewhat related title of "The Psychology of Everyday Things". Reason's book is somewhat tedious in its presentation, but I think it serves to lay a foundation of the vocabulary of the discipline. He carefully defines words like "error", "mistake", and "slip" as they will be used in the Human Factors literature. I enjoyed Kern's descriptions of various accidents, but I didn't derive much value from his prose between accident descriptions.

The problem with slips, mistakes and errors is that most crews don't care to drill down that far. They only want good simple useful tools. And a lot of presentations can become sort of self-absorbed. This does not mean, however, that one should shy away from discussing and explaning cognitive capture, dissonance, biases (confirmation, attribution, etc). And quite odd, in all the years of ground schools and classes, NEVER one on critical thinking and decision making strategies. ???

We have proven we can not train out errors, can't make enough swiss cheese to defense errors, can't write enough rules. We are still dealing with what some, including me, refer to the Mark 1 Human. (this is not to say we don't need definitions, rules and/or training because we do but after we establish someone can do a V1 cut and a coupled Cat IIIB, do we really need to keep checking that every six months?)

Human Factors concepts understand and accept this; rather than trying to change the people, they advocate changing the task so that it becomes doable by non-exceptional individuals. That's one reason I don't have much sympathy for those who strongly advocate mandatory spin training for pilots; more training has very diminishing returns. Better to change the task so that the unattainable level of skill isn't necessary.

IF I hand you the wrong tool or the wrong policy/procedure, you WILL fail at the task at some point. One problem is creating the correct 'tool kit'.

Men...

Women...

(that one is going to get me some flak...)




I finished reading the first paper you linked to; one thing I noticed is that the list of recommendations at the end didn't seem to have any research associated with them to demonstrate that they were effective in solving the problems they were targeted towards. I can easily imagine such experiments, so I'm sure that a professional with academic training in the field should be able to do so.

I encourage students to "decouple" airplane configuration from ATC interactions. That means getting the airplane configured early in the approach, rather than waiting to the FAF. This isn't the most efficient airplane management approach, but, as the article pointed out, while coupling is more efficient, decoupling is more reliable.

Boeing, early on, touted to use the highest level of automation available. That was later revised to mode most appropriate. And most airlines have adopted common arrival configurations sequences with approach flaps on the base or 45 to intercept. next flap setting on final. gear at 1 dot on the ILS and gear down at half dot (or something similar). This spreads workload out, lowers task loading and creates a repeating environment that becomes standard. But it boils down to understanding what is normal and routine and adapting to the changes.

Steve Swauger at Southwest published a paper a number of years ago about how experts perform like novices at times when faced with new challenges. Here in the western world and with the airlines, much of the ops are the same. ATC from gate to gate with vectors to an ILS. And we continue to see hot, high unstablized approaches. Which brings us to a question..

How many go-arounds are allowed? ... should be allowed?

 

[shdw]

Re: Combinations of RPM & Manifold Pressure

Orange and tgray: It has been a pleasure reading your responses on this subject. I particularly agree with making the procedure fit the people, not the people fit the procedure.

Taylor, you might want to add "Flight Discipline" by Kern to your reputar. It was the book we used for my college studies on discipline. He discusses disciplines through just about any flight regime or task you can imagine, some include: automation, communication, regulatory, organizational, personality factors, and a chapter on chaos theory which I found to be a wonderful read.

The other book I haven't seen mentioned is "Basic Flight Physiology" by Richard O. Reinhard. I'm sure things are missed, but all in all it is a wonderful, relatively short (<300 pages), book that covers a white array of human physiological factors and how they effect flight.

 

[Orange Anchor]

Re: Combinations of RPM & Manifold Pressure

SHDW, you might enjoy Sway, The Irresistable Pull of Irrational Behavior.

Also, March's book, A Primer on Decision Making.

Also, Welch's Decisions, Decisions.

Dormer's Logic of Failure.

Klein's Sources of Power and Power of Intuition.

(there goes the summer )

 

[MikeD]

Re: Combinations of RPM & Manifold Pressure

For me, the wisdom underlying Human Factors concepts is that it steps away from the macho attitude that the solution to human error is more training or getting rid of those who can't measure up. While those things have a role, there is a limit to how good human beings can be. Human Factors concepts understand and accept this; rather than trying to change the people, they advocate changing the task so that it becomes doable by non-exceptional individuals. That's one reason I don't have much sympathy for those who strongly advocate mandatory spin training for pilots; more training has very diminishing returns. Better to change the task so that the unattainable level of skill isn't necessary.

You have to quantify where it is that you're trying to apply your theory here, because there are places where you simply can't. Example: military flying. There's nothing "macho" about washing out those that don't or can't measure up; it's simply a way of keeping them and possibly people flying along or with them alive. Pure and simple. There isn't a way to "change the task" so it becomes easier, because (especially for fighter aircraft or other tactical aircraft, helos included) the tasks are designed to insure that people are pushed to their limits and whether or not they cross the boundary of can hack it or can't hack it. And that further translates to many different mission tasks that can't be trained for, but there has to be confidence (both of the pilot, as well as the military) that the pilot can manage them quickly, efficiently, and safely. That confidence comes his getting pushed hard in training, and their being standards that he has to meet, not standards that come down and meet him.

I realize that you only have GA experience, but I'd caution in broadbrushing different teaching methods. What you advocate would work fine for the average Joe GA pilot, where they can determine how difficult they want to make things or what their personal limits are, or whether they even fly that particular day, generally speaking. Not so in other areas of flying. What you falsly term as a "macho attitude" of getting rid of the weak swimmers, is anything but, in this case.