UK Air Traffic Control System Failure - A “one in 15 million” event

[toucana]

A report by NATS (National Air Traffic Services) says that a catastrophic failure of the UK air traffic control system on the August 28 Bank Holiday this year was a caused by a “one in 15 million’ software failure event.

https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf

According to this report, a key sub-system called FPRSA-R (Flight Plan Reception Suite Automated) was thrown into a fail-safe shutdown mode when it attempted to process a flight plan submitted by an un-named airline which included two identically named (but geographically distinct) waypoint markers. The back-up system which runs the same software shut down as well.

During the 4 hours it took to identify and resolve the issue, flight plans across the UK had to be processed manually, lowering the number that could be handled to just 60 per hour - instead of a normal 400. Around 1500 flights had to be cancelled on the Monday alone, and knock-on effects lasted for several days more.

The ICAO (International Civil Aviation Organization) and other bodies have been trying to eradicate the use of non-unique waypoint names, but duplicates do exist around the world. Latest standards state that identical designators should be geographically widely spaced apart. But long-haul flight plans may include duplicates.

In this instance both of the waypoints were located outside of the UK, one towards the beginning of the route, and one towards the end; approximately 4000 nautical miles apart

[Carrock]

12 hours ago, toucana said:

A report by NATS (National Air Traffic Services) says that a catastrophic failure of the UK air traffic control system on the August 28 Bank Holiday this year was a caused by a “one in 15 million’ software failure event.

From the report, a single software failure in calculating a flight path was designed to cause a fatal exception i.e. crash the whole system rather than e.g. generate a 'NOT VALID FLIGHTPATH -MANUAL INTERVENTION REQUIRED' warning for controllers. The backup system must have been designed solely for hardware failure since as soon as it was enabled it experienced the same software failure and crashed.

This particular problem has been fixed so crashing the system was always an unnecessary requirement in handling this safety critical information.

The claim that this failure, after 15 million successful flight plans, is a  ''“one in 15 million" software failure event' implies that there are no more unintended fatal exceptions in the software.

Really?

Fortunately the whole system is now getting a very necessary upgrade.

[swansont]

2 hours ago, Carrock said:

The claim that this failure, after 15 million successful flight plans, is a  ''“one in 15 million" software failure event' implies that there are no more unintended fatal exceptions in the software.

Also that this is a statistical issue and not systematic, i.e. that a similar repetition of waypoints would not cause failure, and that doesn’t ring true.

[toucana]

On 9/7/2023 at 12:29 AM, swansont said:

Also that this is a statistical issue and not systematic, i.e. that a similar repetition of waypoints would not cause failure, and that doesn’t ring true.

I can't help wondering if it was a one-off special charter, (possibly a military cargo flight shipping weapons to Ukraine ?) which generated this malformed flight plan ?. Statistically speaking, it had to be quite a long-haul transatlantic flight to be capable of inadvertently snagging navigational waypoints with identically named designators (given that theses are supposed to be geographically widely separated) - but if it was a regular scheduled passenger flight plan, then surely the same software failure would have happened many times before ?

[swansont]

2 minutes ago, toucana said:

I can't help wondering if it was a one-off special charter, (possibly a military cargo flight shipping weapons to Ukraine ?) which generated this malformed flight plan ?. Statistically speaking, it had to be quite a long-haul transatlantic flight to be capable of inadvertently snagging navigational waypoints with identically named designators (given that theses are supposed to be geographically widely separated) - but if it was a regular scheduled passenger flight plan, then surely the same software failure would have happened many times before ?

Will the next one be inadvertent?