Pilots & Ops folks—What’s your honest take on SMS tools?

[twrightflyer]

Hey folks—I'm a former Army pilot and now work in aviation consulting. I'm considering building a lightweight, modern SMS platform for smaller Part 91 and 135 operators.

Before writing a single line of code, I’m gathering input from pilots, ops directors, and safety managers to understand what really works and what doesn’t in your current system.

Survey link (3 minutes, anonymous) or copy/paste: Part 91/135 Operator SMS Survey

I’ll gladly share back a summary of insights for anyone curious.

Really appreciate any time you can give!

 

[SteveC]

[moved to appropriate forum]

 

[TUCKnTRUCK]

We are a 91 “Small” department as you will.

I’ll be 100% honest, and SMS system which is not directly tied to our dispatch and planning software is not of interest to us. We have an SMS/TEM program but most off the shelf programs require more positions than we have for effective management.

I anticipate AI analysis of our QAR/DFDR tied to sms/ reporting will eventually be a thing, but until then it often feels like the work involved doesn’t justify the returns honestly.

 

[Wardogg]

Hey folks—I'm a former Army pilot and now work in aviation consulting. I'm considering building a lightweight, modern SMS platform for smaller Part 91 and 135 operators.

Before writing a single line of code, I’m gathering input from pilots, ops directors, and safety managers to understand what really works and what doesn’t in your current system.

Survey link (3 minutes, anonymous) or copy/paste: Part 91/135 Operator SMS Survey

I’ll gladly share back a summary of insights for anyone curious.

Really appreciate any time you can give!

Here's your competition. This is what we use.

Nimbl

Nimbl

my.gonimbl.com

 

[ppragman]

Here's your competition. This is what we use.

Nimbl

Nimbl

my.gonimbl.com

How much does this cost you guys?

I mean, like, this looks expensive, and frankly, not very good...

 

[ppragman]

Hey folks—I'm a former Army pilot and now work in aviation consulting. I'm considering building a lightweight, modern SMS platform for smaller Part 91 and 135 operators.

Before writing a single line of code, I’m gathering input from pilots, ops directors, and safety managers to understand what really works and what doesn’t in your current system.

Survey link (3 minutes, anonymous) or copy/paste: Part 91/135 Operator SMS Survey

I’ll gladly share back a summary of insights for anyone curious.

Really appreciate any time you can give!

I don't fly anymore (I also write code for money), but I found most of the ops software and SMS tools were either too general or too bespoke to some other customer's needs. I don't really think there's a "one size fits all" solution to this problem because so many operations are so different. If it were me (and I guess maybe it should be), I'd be building these tools for individual operations.

"Hey, let me build you this, I'll build it the way your operation works, and when I am done, the only monthly costs will be to host it." Some of these products are hundreds or even thousands of dollars per month. That's absurd for what most of them are offering.

 

[killbilly]

I don't fly anymore (I also write code for money), but I found most of the ops software and SMS tools were either too general or too bespoke to some other customer's needs. I don't really think there's a "one size fits all" solution to this problem because so many operations are so different. If it were me (and I guess maybe it should be), I'd be building these tools for individual operations.

"Hey, let me build you this, I'll build it the way your operation works, and when I am done, the only monthly costs will be to host it." Some of these products are hundreds or even thousands of dollars per month. That's absurd for what most of them are offering.

Just thought, since I used to impersonate product managers from time to time in my sales engineering days...

Instead of building for individual operations - which will inevitably suffer from scope creep - what if you built a modular toolset? In other words, give them tools to build their own. I've got a hazy idea of a set of features built on whatever APIs are out there and let them Lego their way into something that works for them.

 

[TUCKnTRUCK]

Just thought, since I used to impersonate product managers from time to time in my sales engineering days...

Instead of building for individual operations - which will inevitably suffer from scope creep - what if you built a modular toolset? In other words, give them tools to build their own. I've got a hazy idea of a set of features built on whatever APIs are out there and let them Lego their way into something that works for them.

eh- in theory sure, but once you go through an Is-bao audit for compliance you end up getting sold on the same product that everybody else has. Ideally SMS/TEM etc is based on a global best practices/safety criteria.

not a huge help if you let me build say “airport/weather” threats into my SMS- but then pull out schedule/duty/circadian disruption. (Yes I’m only talking about parts of a FRAT) but same idea to the larger legos if you will.

 

[Wardogg]

How much does this cost you guys?

I mean, like, this looks expensive, and frankly, not very good...

I have no idea what we pay for it. Above my pay grade.

It has some holes and some things I don't like for sure. But it handles FRATS and ASAP and Hazard/Incident reports well enough for our operation.

 

[ppragman]

I have no idea what we pay for it. Above my pay grade.

It has some holes and some things I don't like for sure. But it handles FRATS and ASAP and Hazard/Incident reports well enough for our operation.

I should start building these for people... I bet I can be a lot cheaper than these kinds of guys and do it custom for their operation, all of the ones I've used have been trying to do too much it felt like? Maybe someone of them were good, but it always felt like they were trying to do everything instead of one or two things that I needed well. Also, the numbers in most of the frats were just totally pulled out of people's ass lol.

 

[ppragman]

Just thought, since I used to impersonate product managers from time to time in my sales engineering days...

Instead of building for individual operations - which will inevitably suffer from scope creep - what if you built a modular toolset? In other words, give them tools to build their own. I've got a hazy idea of a set of features built on whatever APIs are out there and let them Lego their way into something that works for them.

Love this idea. You'd have to be smart about it so that (because of scope creep again) you didn't cover every usecase... but like, getting an operation something like 80% of the way there with boilerplate stuff and being like, "oh, you want to add X? Well, I can put that together from these other two modules."

 

[ppragman]

eh- in theory sure, but once you go through an Is-bao audit for compliance you end up getting sold on the same product that everybody else has. Ideally SMS/TEM etc is based on a global best practices/safety criteria.

not a huge help if you let me build say “airport/weather” threats into my SMS- but then pull out schedule/duty/circadian disruption. (Yes I’m only talking about parts of a FRAT) but same idea to the larger legos if you will.

The FRATs I've used have all been kind of garbage except for one company I worked at to be honest... Every place I worked had some scoring system except for one and there was no actual reason for the scores under the hood, it was kind of funny...

The place that did it right broke the FRAT down by each phase of flight down and looked at the risks in each segment. It was really cool, because let's say the weather sucked at the departure airport, but everything around the departure was fine, why should that drag down the rest of the flight? If it was borderline, you'd note it in the FRAT, call the CP or DO or their designee, "hey man, the weather sucks here, and if I leave I won't be able to get back in, but if something happens there's like 6 airports within 20 minutes, and the weather is good for the rest of the flight?" They'd ask a few questions, then either choose to accept your strategy or not. You didn't end up with really bizarre situations where the flight was moderately sketchy the whole time because you were borderline the whole way (if you needed too many mitigations, it was an automatic scrub), and then also like if you were having a bad day for something, there was a mechanism to put a guy on easier trips. "Bob has flown 40 hours this week, it's been bananas, he's fatigued - ok, he can't accept any mitigation then."

I liked this a lot better than a points system where the numbers were clearly made up by how scary whoever made the FRAT thought that particular thing was but you could end up with a crazy situation where you'd be mashing ice on the whole flight, land at mins at both airports, have to hand deice from some shack, are flying in your region of circadian low, and are 2 points away from scrubbing but there's a thunderstorm 40 miles away from you, moving away from you on a clear VFR day and you have to add +6 or whatever because the "convective weather within 100NM? - +6" or whatever.

As far as I can tell though, literally none of the FRATs I have ever used (and in my career I used maybe 10 different methods for mitigating risk?) had any basis in any statistics.

 

[TUCKnTRUCK]

The FRATs I've used have all been kind of garbage except for one company I worked at to be honest... Every place I worked had some scoring system except for one and there was no actual reason for the scores under the hood, it was kind of funny...

The place that did it right broke the FRAT down by each phase of flight down and looked at the risks in each segment. It was really cool, because let's say the weather sucked at the departure airport, but everything around the departure was fine, why should that drag down the rest of the flight? If it was borderline, you'd note it in the FRAT, call the CP or DO or their designee, "hey man, the weather sucks here, and if I leave I won't be able to get back in, but if something happens there's like 6 airports within 20 minutes, and the weather is good for the rest of the flight?" They'd ask a few questions, then either choose to accept your strategy or not. You didn't end up with really bizarre situations where the flight was moderately sketchy the whole time because you were borderline the whole way (if you needed too many mitigations, it was an automatic scrub), and then also like if you were having a bad day for something, there was a mechanism to put a guy on easier trips. "Bob has flown 40 hours this week, it's been bananas, he's fatigued - ok, he can't accept any mitigation then."

I liked this a lot better than a points system where the numbers were clearly made up by how scary whoever made the FRAT thought that particular thing was but you could end up with a crazy situation where you'd be mashing ice on the whole flight, land at mins at both airports, have to hand deice from some shack, are flying in your region of circadian low, and are 2 points away from scrubbing but there's a thunderstorm 40 miles away from you, moving away from you on a clear VFR day and you have to add +6 or whatever because the "convective weather within 100NM? - +6" or whatever.

As far as I can tell though, literally none of the FRATs I have ever used (and in my career I used maybe 10 different methods for mitigating risk?) had any basis in any statistics.

That’s because data costs money, and, nobody wants to pay for access/ so much data is stuck behind gatekeepers (cough 121, alpa etc)

We also have a remarkably low incident per flight hour rate- so hard “risk” data is hard to come by.

QAR’s can give you what your looking for- but it’s hard to develop trends in your own department if you only fly 100-200 hours a year.

Get some 50,000 hours of flying a month, and you can correlate. It gets a lot easier to look at all landings that were say- out of the touchdown zone, and then pull casuals like weather, winds, fatigue, airport familiarity etc.

Ideally with SMS the fray scores get adjusted as risks are identified… but there’s never enough feedback. The number of pilots that will drop a risk/threat/SMS report when they make a mistake because of small factors like an early/ late day is slim. Nobody wants to do the paperwork etc. they just want to park the plane and go home.

Regardless IMHO FRATS should mostly serve to identify risks- and make the crew aware + give them a solid and informed reason to scrub a flight. A frat should never be used to justify why a flight can go.

 

[WingedCelt]

ppragman nailed it. You have both ends of the spectrum with very little in the way of in between. Were I building the SMS programs, I d make it boilerplate and able to modularize the system as it were-dont have a dispatch department? No need for their inputs or ASAP participation. FRATS too generic? One HondaJet operator there was no availability for us to really manipulate for crosswinds, which in that airplane, absolutely demanded it because max crosswinds meant for a very risky flight.

It all depends-were I you, look at it like you were buying your first jet-what do you need, what do you want, and how can we tailor make a system that works for you and your department.

 

[Minuteman]

The DOT Transportation Safety Institute has a several Army rotorcraft pilots in their ranks and would probably be happy to at least chat. Informally and unofficially I got the impression that Baldwin was a competent offering.

I think scope creep is good when starting from nothing. Start doing one thing and get good at all facets of that: built it, meet SLAs for hosting it, support the users, and figure out what needs exist. A kind of spiral development, but don't be all formal & weird about it.

I have seen implementations at large organizations that are the modular platforms, and many (most?) people hate it because that kind of thing started life as an adaptation of something else for a new use ("hey our warehouse inventory software is now a process excellence tool!"). Business processes get contorted to limitations of the platform. And those limitations seem to exist for two big reasons: a) it is always beyond the budget to get a polished version of someone's vision, or b) there is little vision beyond process continuity and optimizing the same thing for leadership praise.

One airline I am aware of is in the process of switching platforms and so far they have spent $Texas to hire consultants to perform a lift-and-shift implementation on the new software vendor's platform with changes to address the accumulated criticism of the old platform still-pending. From a vendor's perspective it's probably a great business model, unless the airline customer thinks they're ever going to be "done" and stop budgeting for evolution (this always happens).

 

[ppragman]

ppragman nailed it. You have both ends of the spectrum with very little in the way of in between. Were I building the SMS programs, I d make it boilerplate and able to modularize the system as it were-dont have a dispatch department? No need for their inputs or ASAP participation. FRATS too generic? One HondaJet operator there was no availability for us to really manipulate for crosswinds, which in that airplane, absolutely demanded it because max crosswinds meant for a very risky flight.

It all depends-were I you, look at it like you were buying your first jet-what do you need, what do you want, and how can we tailor make a system that works for you and your department.

I am building a bespoke tool right now for a friend I used to work for right now. I'm not doing their FRAT but I'm basically rebuilding a weight and balance spreadsheet I made for them like 10 years ago, but I'm doing it with a progressive web app rather than excel. That said, I'd really like to build the "ultimate" FRAT. I'd like to do something like this: where every flight, log data about the flight, weather, airplane status/MELs/etc, normal FRAT data entry, and so on. Then after the flight the pilot logs "how sketchy it felt" on a scale of 1 to 5. Nothing crazy, but you throw that into the mix. Then eventually (if you don't try to change the FRAT aggressively ever 2 hours) you get something where you can roughly predict how sketchy you expect a flight to be based on the data you have. You can also project all the data you have collected up into some high dimensional space and measure "how unusual" a flight is too. After a year or two, you could actually probably analytically predict terrible flights before they happen, which would basically allow you to change your behavior pre-emptively. I think that would be super useful...

I don't know, I just have to get someone to let me build it is the problem lol.

 

[killbilly]

I am building a bespoke tool right now for a friend I used to work for right now. I'm not doing their FRAT but I'm basically rebuilding a weight and balance spreadsheet I made for them like 10 years ago, but I'm doing it with a progressive web app rather than excel. That said, I'd really like to build the "ultimate" FRAT. I'd like to do something like this: where every flight, log data about the flight, weather, airplane status/MELs/etc, normal FRAT data entry, and so on. Then after the flight the pilot logs "how sketchy it felt" on a scale of 1 to 5. Nothing crazy, but you throw that into the mix. Then eventually (if you don't try to change the FRAT aggressively ever 2 hours) you get something where you can roughly predict how sketchy you expect a flight to be based on the data you have. You can also project all the data you have collected up into some high dimensional space and measure "how unusual" a flight is too. After a year or two, you could actually probably analytically predict terrible flights before they happen, which would basically allow you to change your behavior pre-emptively. I think that would be super useful...

I don't know, I just have to get someone to let me build it is the problem lol.

How do you account for wide variability in the subjective nature of the pilot's 1-5 rating system? I've flown with guys where I'd rate a flight a "4" (this was sketchy AF) and they'd be like, "Aw, that was a 1-2."

 

[Wardogg]

How do you account for wide variability in the subjective nature of the pilot's 1-5 rating system? I've flown with guys where I'd rate a flight a "4" (this was sketchy AF) and they'd be like, "Aw, that was a 1-2."

An approach at night in weather in mountainous terrain is riskier than day VFR into the flatlands. Irrespective of pilot perception. A properly built FRAT captures and highlights those risks.

 

[ppragman]

How do you account for wide variability in the subjective nature of the pilot's 1-5 rating system? I've flown with guys where I'd rate a flight a "4" (this was sketchy AF) and they'd be like, "Aw, that was a 1-2."

You'd have to consider the pilot too and you'd have to have enough data.

This sort of thing probably wouldn't work for a shop with only 3 guys or whatever? But if you had enough crews you'd probably be alright. The analogy I can think of is recommender systems. Spotify is really good at predicting what I will like musically - the methodology would be similar here.

Now, of course if you get a guy who always marks flights as really sketchy, it might be weird, but yeah. The other thing that's worth mentioning is that it might not work. But I think it would.

An approach at night in weather in mountainous terrain is riskier than day VFR into the flatlands. Irrespective of pilot perception. A properly built FRAT captures and highlights those risks.

Well... mostly. Flat light has entered the chat. How many 207s have drifted into the tundra due to that over the decades? But beyond edge cases, you are correct, and theoretically, you'd already have that data in the tool prior in the model so really the "1-5 score" is kind of a measure of "how surprising" the flight was given what you already knew. So even though "Billy Badass" rates the flight as "1-2" the model already has the airports which themselves have some data on them about the approaches, etc.