AVSIG: FAA's many soft spots wwswsigarch.jpg (7236 bytes)

✈ . . . . . . ✈ . . . . . ✈ . . . . ✈ . . . ✈ . . ✈ . ✈ . . ✈ . . . ✈ . . . . ✈ . . . . . ✈ . . . . . . Touch-and-Go to our Live Forum (This is a Read-only Archive of the 2004-2017 AVSIG Forum)


AVSIG Discussion Sections >> FAA Topics

Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | >> (show all)
Ward Miller POU-NY
Top Gun


Reged: 05/05/04
Posts: 10508
Loc: New York
FAA's many soft spots
      #397129 - 09/29/14 01:11 PM

There has been a lot of media coverage about the thousands of cancelled
flights and stranded passengers recently in the Chicago area, but I have yet
to see any significant coverage of how the FAA is designed so *one* person
with some *simple* actions can bring the system to it knees.

What is the action plan if the ORD tower suddenly went up in a puff of
smoke? Stuff can happen! That one is sort of easy to plan for, but do they
have a plan and is it known to all who would have to react?

What are their plans if a Center suddenly goes down? Apparently, they have
none. Gasoline from a nearby auto accident could spill down a manhole, catch
fire and sever all the landline circuits. They have a plan when one or two
positions go down, but what about the whole facility going Tango Uniform?

I just sit here and shake my head. How could one person cause such havoc? I
suspect there are hundreds of such soft spots throughout the FAA's air
traffic control system. How could they obviously never have planned for such
simple yet major disruptions?

"CODE RED!! IMPLEMENT EMERGENCY PLAN B2!!" Only in our dreams. . .


Post Extras: Print Post   Remind Me!   Notify Moderator  
Russell Holton
AVSIG Member


Reged: 07/07/05
Posts: 14136
Re: FAA's many soft spots [Re: Ward Miller POU-NY]
      #397131 - 09/29/14 01:30 PM

What is "the system"? The airport or the overall airspace?

Any number of things can close one airport. I suspect the real problem is the airlines may not have a backup plan if their hub goes down or is drastically reduced in capacity. Is that the FAA's fault or the airlines?

Edit: No sooner than I post when I hear this on the radio:

FAA Orders Security Review After Sabotage Draws Scrutiny

"Of 29 racks of computers driving the communications equipment, 20 were destroyed by fire and water damage, FAA Administrator Michael Huerta said today."

Really? Water for an equipment fire?? Bad idea.

But I suspect the real issue will be a single individual being unsupervised in such a sensitive area.






Post Extras: Print Post   Remind Me!   Notify Moderator  
John O'Shaughnessy [FCM]
Top Gun


Reged: 09/13/01
Posts: 5059
Loc: Minnesota
Re: FAA's many soft spots [Re: Ward Miller POU-NY]
      #397134 - 09/29/14 02:33 PM

I ask myself the same question.

We do a large number of Business Impact Assessments for our clients. Based on the criticality of their technical infrastructure, we give recommendations on how to achieve an appropriate level of resiliency in their environments.

For example, if a client suggests that 4 hours of downtime could cost them a million dollars, we recommend a fully redundant system, tested regularly (that is key). That redundant system might cost 10 million, but if their own internal requirements are never to have 4 hours of downtime, that is what you design to.

It would appear that the Chicago Center, (and, I fear, all the rest) have not been built with total facility failure measuring into days as something to realistically plan around.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Russell Holton
AVSIG Member


Reged: 07/07/05
Posts: 14136
Re: FAA's many soft spots [Re: John O'Shaughnessy [FCM]]
      #397136 - 09/29/14 03:30 PM

Quote:

For example, if a client suggests that 4 hours of downtime could cost them a million dollars, we recommend a fully redundant system, tested regularly (that is key). That redundant system might cost 10 million, but if their own internal requirements are never to have 4 hours of downtime, that is what you design to.




Which begs the question - what was the FAA's design criteria? Given they are operational so quickly, I think they had to have some kind of plan and resources in place.


Post Extras: Print Post   Remind Me!   Notify Moderator  
B. Butler (Oregonian)
Top Gun


Reged: 05/15/04
Posts: 9760
Loc: Ashland, Oregon
Re: FAA's many soft spots [Re: John O'Shaughnessy [FCM]]
      #397137 - 09/29/14 03:58 PM

Quote:

we recommend a fully redundant system, tested regularly (that is key). That redundant system might cost 10 million,




You and Ward are over-simplifying:

The level of redundancy you seek is several orders of magnitude greater; perhaps $50 billion<?> and the vulnerability isn't really the infrastructure, which is largely redundant already.

FAA digitized its Center RADAR systems and moved their "data path" off of mountain-top microwave systems in the mid-70s, by 1990, they were able to direct radar data to adjacent centers in the event of emergencies. Theoretically, Chicago Center could be worked from Indy or Oberlin, or Auburn, WA, for that matter. There are only two things preventing that. One is the lack 50 or 60 installed but unused radar sector suites, and the other is the people to man them.

I can assure you that every air traffic facility has a "Continuity of Service" letter-of-agreement with the adjacent facilities and with the contained approach controls also, but if you think they can call a couple dozen guys out of the lunch room and the training office and run a 200-arrival-hour into O'Hare, 45 into MDW, 50 into MKE-airports and procedures with which they have only a minimal awareness-and 400 or so overflights per hour, you really don't understand the complexity of the system.

The adjacent facilities will be able to handle a significant fraction of the overs, and reroute many around the airspace, especially for a small center like ZAU, but the arrival capacity is going to fall to something on the order of 30%, and departures are simply out of the question.

Much of the system complexity operates outside-let's say adjacent to-the air traffic system. If capacity at a hub falls to-oh, say...50%, it actually approaches zero, because the airlines are bringing pilots, attendants, crews and passengers from disparate locations, and counting on all of them to arrive, shuffle, and depart. Drop one factor from that flight, and it gets cancelled, even if the air-traffic system can provide a departure opportunity.

As for fire-fighting, just what agents would you recommend for a space which includes both electronic devices and large numbers of people? I was at COS tower one day when they triggered the new halogen system and nearly asphyxiated a whole crew.

Be assured, the FAA is not stupid, they worry about this sort of thing every day, but the system they operate with a remarkable level of fidelity has a complexity which boggles the imagination of most of us.

--------------------
"Why not be a nihilist? A man has to believe in something."
-Bernie Gunther

Edited by B. Butler (Oregonian) (09/29/14 04:45 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
sreyoB yrraL
AVSIG Member


Reged: 05/16/04
Posts: 9442
Re: FAA's many soft spots [Re: Russell Holton]
      #397138 - 09/29/14 04:22 PM

Quote:

Any number of things can close one airport. I suspect the real problem is the airlines may not have a backup plan if their hub goes down or is drastically reduced in capacity. Is that the FAA's fault or the airlines?



The airlines have a plan. It is to cancel affected flights, regroup, and restart operations when the crisis is over. There really is no other viable alternative.

Next time you're at a large hub look around at all the people, facilities, and equipment that it takes to operate a hub. You can't just pick that all up and move it to another large, unused, airport--as if one existed--for a day or two.

The push, and resultant legislation, over the past few years to reduce ground delays has provided significant incentive to shut down the operation sooner rather than later. In the past, airlines would continue to attempt to operate as many flights as possible in a reduced, yet unreliable, schedule. That no long happens. Now they shutdown, put all crews into rest, and organize a recovery plan for when operations resume. The downside is that it may take the better part of a week to get all of the displaced passengers reaccomodated.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Russell Holton
AVSIG Member


Reged: 07/07/05
Posts: 14136
Re: FAA's many soft spots [Re: B. Butler (Oregonian)]
      #397140 - 09/29/14 04:33 PM

Quote:

As for fire-fighting, just what agents would you recommend for a space which includes both electronic devices and large numbers of people?




In terms of total damage, you might do well with simply killing power and let it burn itself out.

Technically, "electrical fires" are not fires. They just create enough heat for something else to burn - including things that are not normally combustible. (Including certain kinds of fire retardants.) Kill the power, and you kill the heat source. Typical computer room environment doesn't have much in the way of combustibles so the fire should be self-limiting.

Now, in this situation an gasoline was used, so the fire is going to be nasty. But it appears that CO2 would do well in this situation.

But going into an equipment room with fire hoses and I have to ask: Did the fire suppression cause more damage than the fire was capable of?

But it also raises the question: Since this seems to be an equipment room with few people in it, what kind of fire suppression system did it have?


Post Extras: Print Post   Remind Me!   Notify Moderator  
Ward Miller POU-NY
Top Gun


Reged: 05/05/04
Posts: 10508
Loc: New York
FAA's many soft spots [Re: B. Butler (Oregonian)]
      #397148 - 09/29/14 06:29 PM

>> You [John O'Shaughnessy] and Ward are over-simplifying. . . <<

You betcha! I suspect John was talking about a job similar to one I held.
We got paid (or not) for results. I've managed systems where an interruption
was not allowed. They are possible and don't have to cost the equivalent of
the national debt.

In this case it appears the major problem was a bunch of adjacent processors
were damaged by the fire. They could have been in a containable environment
so it would not spread. They also could have had redundant processors to
take over.

I say again, when one guy without any sophisticated equipment could affect
millions of passengers and cost untold millions of dollars, something basic
in the system design was missing.

>> Be assured, the FAA is not stupid, they worry about this sort of thing
every day, but the system they operate with a remarkable level of fidelity has
a complexity which boggles the imagination of most of us. <<

If they worried about it, did they have an emergency plan to cover a
situation like this? You might be surprised but there are hundreds of
enterprizes just as critical and just as complex, in their own way, as is the
FAA. Years ago I was intimately involved with two of them and they had
numerous emergency plans, some of which had to be called upon occasionally.

If the terrorists learned from this, we could be in very serious trouble.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Russell Holton
AVSIG Member


Reged: 07/07/05
Posts: 14136
Re: FAA's many soft spots [Re: Ward Miller POU-NY]
      #397150 - 09/29/14 07:11 PM

Quote:

In this case it appears the major problem was a bunch of adjacent processors were damaged by the fire. They could have been in a containable environment so it would not spread. They also could have had redundant processors to take over.




That too, but it also sounds like a case of problems of co-located communication lines. I'm sure everyone considered single-point failure of individual boxes, but I'm not sure as they considered single-point failure at the room level.

At some point you have to consider single-point failure of the entire tower. And that may be what mode they're running in now.

Perhaps a room-level fault was considered too remote.


Post Extras: Print Post   Remind Me!   Notify Moderator  
sreyoB yrraL
AVSIG Member


Reged: 05/16/04
Posts: 9442
Re: FAA's many soft spots [Re: Russell Holton]
      #397152 - 09/29/14 08:07 PM

Quote:

Perhaps a room-level fault was considered too remote.



Or the level of backup was too costly.

Several times per year you have disruptions of similar scale from severe weather events. At some point it becomes cheaper to just shut down for a day, or so, then it does to have a sufficiently robust backup. Especially true when such an event does not happen with any regularity as does the weather events.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | 4 | 5 | 6 | 7 | >> (show all)



Extra information
0 registered and 40 anonymous users are browsing this forum.

Moderator:  Mike Overly 

Print Topic

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled

Rating:
Topic views: 14506

Rate this topic

Jump to

Contact Us AVSIG

Powered by UBB.threads™ 6.5.5

Logout   Main Index    AVSIG Aviation Forum