Welp, that sucks...

[Hacker15e]

I call B.S.

How can one aircraft at a given altitude do that to the backup system? There is a lot more to this story is my guess.

There's nothing shady going on here, actually, either with the USAF or the FAA:

 

[Hacker15e]

the flight plan for the U2 had too many elements and the ERAM system simply ran out of memory.

That's not what the memo I saw from the FAA said (see above).

I actually watched this mission get planned and the DD175 flight plan get written by a coworker of mine. Can you elaborate on the "too many elements" part, if that was indeed a component of the problem?

 

[TripSix]

There's nothing shady going on here, actually, either with the USAF or the FAA:

Thanks for posting this. I didn't have the stones to drop the big "NOPE" and post it myself.

 

[Ozelot]

That's not what the memo I saw from the FAA said (see above).

I actually watched this mission get planned and the DD175 flight plan get written by a coworker of mine. Can you elaborate on the "too many elements" part, if that was indeed a component of the problem?

Okay ya I didn't see that. What I was told by the ERAM people at my old workplace, so I'm sure the story got mixed up a bit by the time I heard it. They said it crashed after 70 or 71 elements, I dont remeber 100% what number he said. He also stated the system kept trying to process it and it filled up memory and locked up.

Edit: Now that I read that memo. The info I got was from before that memo came out so...who knows what things have transpired since then.

 

[Hacker15e]

Okay ya I didn't see that. What I was told by the ERAM people at my old workplace, so I'm sure the story got mixed up a bit by the time I heard it. They said it crashed after 70 or 71 elements, I dont remeber 100% what number he said. He also stated the system kept trying to process it and it filled up memory and locked up.

Edit: Now that I read that memo. The info I got was from before that memo came out so...who knows what things have transpired since then.

Okay, gotcha.

Again, I didn't write that flight plan, but I saw it written and I know what methodology was used to develop it. If there was something funky about it that made it problematic for the FAA system, that would be quite useful to know.

 

[Bernoulli Fan]

You're both basically right. The buffer for ERAM flightplans is (was; they're increasing it after this incident) 128k. It can be exceeded by too many route elements, as would have been the case with the Dreamliner sketching its logo, or too many altitudes to conflict probe, or a combination, as it was in this case. This is how it was briefed to us yesterday by the chief.

*edit* Kind of a misleading thread title BTW

 

[ahw01]

And the other thread title wasn't?

 

[Ozelot]

You're both basically right. The buffer for ERAM flightplans is (was; they're increasing it after this incident) 128k. It can be exceeded by too many route elements, as would have been the case with the Dreamliner sketching its logo, or too many altitudes to conflict probe, or a combination, as it was in this case. This is how it was briefed to us yesterday by the chief.

*edit* Kind of a misleading thread title BTW

I was working the mid the night he came all over the Dakotas. I had to RS and DM plans all night. Not really a big deal but they obviously knew there was a limit on elements and figured nobody would ever file that many fixes.

 

[Autothrust Blue]

You're both basically right. The buffer for ERAM flightplans is (was; they're increasing it after this incident) 128k. It can be exceeded by too many route elements, as would have been the case with the Dreamliner sketching its logo, or too many altitudes to conflict probe, or a combination, as it was in this case. This is how it was briefed to us yesterday by the chief.

*edit* Kind of a misleading thread title BTW

Most bugs in C/C++ applications are caused by either (1) not having enough memory, or (2) managing that memory incorrectly.

I'm not intimately familiar with the details of ERAM, as when folks I know talk about it I sort of glaze over and reach for my phone, but I am a programmer...and:

Making the buffer larger fixes item (1). Item (2), however, is more concerning. The software isn't really "correct" if the fix is making the buffer 2048K, or any arbitrarily larger size--the correct fix is to bounce the flight plan as "requires manual processing" or whatever it is and NOT OVERRUN THE FRAKKING BUFFER!

 

[ahw01]

Shoulda used Fortran

 

[Bernoulli Fan]

Most bugs in C/C++ applications are caused by either (1) not having enough memory, or (2) managing that memory incorrectly.

I'm not intimately familiar with the details of ERAM, as when folks I know talk about it I sort of glaze over and reach for my phone, but I am a programmer...and:

Making the buffer larger fixes item (1). Item (2), however, is more concerning. The software isn't really "correct" if the fix is making the buffer 2048K, or any arbitrarily larger size--the correct fix is to bounce the flight plan as "requires manual processing" or whatever it is and NOT OVERRUN THE FRAKKING BUFFER!

2048k? You give them too much credit. New buffer will be 384k!

The chief did say that if a flightplan overruns the buffer in the future, it will not be passed to the backup channel, so presumably only the active ERAM channel will fail and it's only a couple button-press to switch. I eagerly await.

 

[Autothrust Blue]

2048k? You give them too much credit. New buffer will be 384k!

The chief did say that if a flightplan overruns the buffer in the future, it will not be passed to the backup channel, so presumably only the active ERAM channel will fail and it's only a couple button-press to switch. I eagerly await.

Bogus data shouldn't be duplicated between redundant systems!

 

[Autothrust Blue]

Shoulda used Fortran

It's 2014--managing your own memory is so 1995.

 

[bronco21016]

The software that crashed is a newly written program, we had tons of growing pains at my old center when we went live with it almost 2 years ago now. I'm surprised it lasted this long without bringing down the system.

As I mentioned before in the other thread, the flight plan for the U2 had too many elements and the ERAM system simply ran out of memory. We encountered a similar situation at my old center and the "workaround" was to make multiple flight plans for the aircraft so this wouldn't happen. ERAM has come a LONG way since the beginning, it used to simply remove a data block form a airplane if it was turned more than 30 degrees. The programmers and the FAA management (non controllers) thought..."Why would they ever need to turn someone more than 30 degrees".

I just went through the thread and realized I hadn't read all of the technical discussion already on here before I made my comment about there being more to the story. Doh!

Thanks for educating me a bit on it. Speaking of education. Anybody have a resource to learn more in-depth how ATC systems work? Would probably help us 121 guys be more patient with things if we understood at least a little bit of what was going on under the hood.

 

[mechan9]

I was working the mid the night he came all over the Dakotas.

Ewwwwww.

 

[Ozelot]

Ewwwwww.

Someone took the bait!

 

[Autothrust Blue]

I just went through the thread and realized I hadn't read all of the technical discussion already on here before I made my comment about there being more to the story. Doh!

Thanks for educating me a bit on it. Speaking of education. Anybody have a resource to learn more in-depth how ATC systems work? Would probably help us 121 guys be more patient with things if we understood at least a little bit of what was going on under the hood.

"Your EDCT is when we bloody well feel like it!"

I kid. I too am interested, particularly in how decisions about delay programs are made (for my job) and how the computer systems work (because I used to do that, too).

 

[Bernoulli Fan]

"Your EDCT is when we bloody well feel like it!"

I kid. I too am interested, particularly in how decisions about delay programs are made (for my job) and how the computer systems work (because I used to do that, too).

I've offered before, but again, if anyone wants an LA center tour, just PM me. For other facilities, find the phone number and call them and I bet they'll find someone to show you around.

 

[Autothrust Blue]

I've offered before, but again, if anyone wants an LA center tour, just PM me. For other facilities, find the phone number and call them and I bet they'll find someone to show you around.

Schlepping out to PMD isn't really something I want to do but let me see about making it a trip with my tech center friend.

 

[fholbert]

Most bugs in C/C++ applications are caused by either (1) not having enough memory, or (2) managing that memory incorrectly.

I'm not intimately familiar with the details of ERAM, as when folks I know talk about it I sort of glaze over and reach for my phone, but I am a programmer...and:

Making the buffer larger fixes item (1). Item (2), however, is more concerning. The software isn't really "correct" if the fix is making the buffer 2048K, or any arbitrarily larger size--the correct fix is to bounce the flight plan as "requires manual processing" or whatever it is and NOT OVERRUN THE FRAKKING BUFFER!