GM1 to AMC1 ATS.OR.205(b)(4) Safety assessment and assurance of changes to the functional system

RISK ANALYSIS IN TERMS OF PROXIES — EXAMPLES Point (c) of AMC1 ATS.OR.205(b)(2) allows safety assessment to be performed in terms of risk, proxies or a combination of risk and proxies. This GM provides two examples to illustrate the use of proxies in safety analysis.

(a)Use of proxies when assessing the safety of a wind farm installation

(1)A wind farm is to be introduced on or near an aerodrome. It is assumed that before the introduction of the wind farm, the safety risk of the air traffic services being provided at the aerodrome was acceptable. To return to this level after the introduction of the farm, the change would also be acceptable. A diagram showing the effects this has on the risk at the aerodrome is shown below:

Figure 1: Evaluation of risks after the introduction of wind farm

(2)The risk due to the introduction of the wind farm will rise from ① to ②, if not mitigated, because:

(i)turbulence will increase and so may destabilise manoeuvring of aircraft;

(ii)the movement of the blades will cause radio interference (communications radio and surveillance radar) and so communications may be lost or aircraft may be hidden from view on the radar screen; and

(iii)the flicker in the peripheral vision of ATCOs, caused by the rotation of the blades, may capture attention and increase their perception error rate.

(3)The problem of analysing the safety impact can be split into these areas of concern since they do not interact or overlap and so satisfy the independence criterion (b) of AMC2 ATS.OR.210(a). However, whilst it can be argued that each is a circumstantial hazard and that in each case a justifiable qualitative relationship can be established linking the hazard with the resulting accident (so satisfying the causality criterion (a) of AMC2 ATS.OR.210(a)), the actual or quantitative logical relationship is, in each case, extremely difficult to determine. Conditions for seeking proxies have, therefore, been established: Performing a risk evaluation using actual risk may not be worthwhile due to the considerable cost and effort involved; and The first two criteria for proxies have been satisfied. Consequently, it may be possible to find proxies that can be used more simply and effectively than performing an analysis based on risk.

(4)The solutions proposed below are for illustrative purposes only. There are many other solutions and, for each change, several should be investigated. In this example, the following proxies, which satisfy the measurability criterion (c) of AMC2 ATS.OR.210(a), are used to set safety criteria:

(i)Turbulence can be measured and predicted by models so the level of turbulence can be a proxy. In this example, let’s assume the only significant effect of turbulence is to light aircraft using a particular taxiway. It is possible to predict the level of turbulence at different sites on the aerodrome and an alternative taxiway is found where the level of turbulence after the introduction of the wind farm will be less than that currently encountered on the present taxiway. This can be confirmed during operation after the change by monitoring.

(ii)Signal quality can be also be predicted by models and measured so it can be used as a proxy. In this example, it is possible to move the communications transmitter and receiver aerials so that communications are not affected by interference. Sites can be found using modelling and the signal quality confirmed prior to moving the aerials by trial installations during periods when the aerodrome is not operating.

(iii)Human error rate in detecting events on the manoeuvring area can be measured in simulations and can be used as a proxy. It is suggested that increasing the opaqueness of the glass in the control tower will reduce the effects of flicker on the ATCOs, but there is no direct relationship between the transmissivity and the effects of flicker. It is, therefore, decided to make a simulation of the control tower and measure the effects of flicker on human error rate using glass of different levels of transmissivity. However, there is a conflict between increasing the opaqueness of the glass to reduce the effects of flicker and decreasing it to improve direct vision, which is needed so that manoeuvring aircraft can be seen clearly. In other words, the simulation predicts a minimum for the human error rate that relates to a decrease, as the effects of flicker decrease, followed by an increase, as the effects of a lack of direct vision increase. This minimum is greater than the human error rate achieved by the current system and so the risk of the wind farm, in respect of flicker, cannot be completely mitigated. This is shown by the red box with a question mark in it on the diagram.

(5)Finally, the argument for the performance of surveillance radars is commonly performed using risk. This can be repeated in this case since the idea is to filter the effects of the interference without increasing the risk. Moreover, if necessary, a system may be added (or a current one improved) to reduce the risk simply and economically and the effects of the additional system may be argued using risk.

(6)Since risks can be combined, the safety impacts of the changes to the surveillance radar by filtering the effects of the interference together with the addition of another system or the improvement of the current system can be established by summing the risks associated with these two kinds of change.

(7)In these circumstances, it is not possible to argue objectively that the risk of introducing the wind farm has been mitigated, as risks cannot be summed with proxies. This demonstrates the difficulties of using proxies. However, it may be possible to argue convincingly, albeit subjectively, that installing another system or improving the current system improves the current level of risk by a margin large enough to provide adequate compensation for the unmitigated effects of flicker.

(8)In summary, this example shows how proxies and risks can be combined in a single assurance case to argue that a change to a functional system can be introduced safely. It also demonstrates that the strategies available to demonstrate safety are not generic, but are dependent on identifying analysable qualities or quantities related to specific properties of the system or service that are impacted by the change.

(b)Use of proxies when changing to electronic flight strips

(1)An air traffic services provider considers the introduction of a digital strip system in one of its air traffic control towers to replace the paper flight progress strips currently in use. This change is expected to have an impact on several aspects of the air traffic control service that is provided such as the controller’s recollection of the progress of the flight, the mental modelling of the traffic situation and the communication and task allocation between controllers. A change of the medium, from paper to digital, might, therefore, have implications on the tower operations, and, hence, on the safety of the air traffic. The actual relation between the change of the strip medium and the risk for the traffic is, however, difficult to establish.

(2)The influence of the quantity on the risk is globally known, but cannot easily be quantified. One difficulty is that strip management is at the heart of the air traffic control operations: the set of potential sequences of events from a strip management error to an accident or incident is enormous. This set includes, for example, the loss of the call sign at the moment a ground controller needs to intervene in a taxiway conflict, and whether this results in an incident depends, for example, on the visibility. This set also includes the allocation of a wrong standard instrument departure (SID) to an aircraft, and whether this results in an accident depends, for example, on the runway configuration. [Figure or form omitted from this preview — available in the Avioverse workspace library.] Figure 2: Notional Bow Tie Model of a strip management error

(3)The Bow Tie Model of a strip management error has, figuratively speaking, a vertically stretched right part. This expresses that a hazard — such as the loss of a single strip — may have many different outcomes which heavily depend on factors that have nothing to do with the cause of the hazard — factors such as the status of the aircraft corresponding to the absent strip, that aircraft’s position on the aerodrome, the traffic situation and the visibility.

(4)Another difficulty with the relationship between the change of the medium and the risk to the air traffic is that several human and cultural aspects are involved. The difficulty lies in the largely unknown causal relationship between these human and cultural aspects and the occurrences of accidents and incidents. As an example of this, it is noted that strip manipulation — like moving a strip into another bay, or making a mark to indicate that a landing clearance is given — assists a controller in distinguishing the potential from the actual developments. The way of working with paper strips generates impressions in a wider variety than digital strips by their physical nature: handling paper strips has tactile, auditory and social aspects. This difference in these aspects may lead to a difference in the quality of the controller’s situation awareness which may lead to a difference in the efficacy of the controller’s instructions and advisories, which may lead to a difference in the occurrence of accidents and incidents. However, the relation between the change of the medium and the risk for the air traffic is difficult to assess and would require a great deal of effort, time and experimentation to quantify. [Figure or form omitted from this preview — available in the Avioverse workspace library.] Figure 3: Relation between the change of flight strip and the risk

(5)There is probably a relation between the change of the flight progress strip medium and the risk for air traffic: a new human–machine interface may have an effect on the situation awareness of some individual controllers in some circumstances, which might have an effect on whether, when and what instructions are given, and this in turn influences the aircraft movements, and, hence, the risks. The question by what amount risks increase or decrease is very hard to answer.

(6)Performing a risk evaluation using actual risk may not be worthwhile due to the difficulties and considerable cost and effort involved in assessing the risk of the change directly. Therefore, the use of proxies might be preferred. A quantity is only considered an appropriate proxy if it satisfies the criteria in point AMC2 ATS.OR.210(a):

(i)Causality: The quantity used as proxy can be expected to be influenced by the change, and the risk can be expected to be influenced by the quantity. In addition to this causal relationship, a criterion can be formulated and agreed upon that expresses by which amount the value of the quantity may shift due to the change. Note that the influence of the proxy on the risk cannot easily be quantified, otherwise it might be more beneficial to use risk as a measure and the quantity as an auxiliary function.

(ii)Measurability: The influence of the change on the quantity can be assessed before as well as after the change.

(iii)Independence: When the proxy selected does not cover all hazards, a set of proxies should be used. Any proxy of that set should be sufficiently isolated from other proxies to be treated independently.

Change Proxy Risk to traffic ! ? Change Proxy Risk to traffic ! ?

Figure 4: Relation between proxy and risk

(7)There is a relationship between the change and the proxy, and there is a relationship between the proxy and the risk to traffic. The first relationship can be assessed (indicated by the ‘!’), while the second cannot (indicated by the ‘?’). An acceptance criterion is typically formulated for the amount the proxy value might increase or decrease.

(8)Proxy 1: Head-down time. The head-down time is a good proxy as it satisfies the conditions of:

(i)Causality: It is known that more head-down time leads to a higher risk but there is no well-established or generally accepted statement in literature in terms of: ‘x % more head-down time implies y% more accidents’, not to mention for the specific circumstances of the specific air traffic control tower. The causal relationship indicated in Figure 4 can be established because:

(A)the head-down time can be expected to change as the manipulation, writing and reading of digital strips might cost more, or perhaps less, attention and effort than the handling of paper strips;

(B)the loss of head-up time of ground and runway controllers implies less surveillance, at least less time for the out-of-the-window-view in good visibility, and this implies a later or less probable detection of conflicts; and

(C)an example of an acceptance criterion reads: ‘The introduction of the digital strip system does not lead to a significant increase in the head down time’.

(ii)Measurability: The influence of the change on the head-down time can be assessed before the change by means of real-time human-in-the-loop experiments in which controllers are tasked to handle equal amounts of traffic in equal circumstances, one time using paper strips and another time using digital strips. The percentage of head-down time can then be determined by observing the controllers by cameras and eye-trackers.

(9)Proxy 2: Fraction of erroneous SID allocations. The fraction of erroneous SID allocations is a good proxy as it satisfies the conditions of:

(i)Causality: It can be imagined that an erroneous SID selected in the flight management system (FMS) might lead to accidents, but the precise conditional probability is small and difficult to estimate as it depends on several external factors such as the flight paths of the correct and incorrect SIDs, the presence of other traffic, the timing and geometry of the trajectories, the cloud base or the vigilance of the controller. The causal relationship indicated in Figure 4 can be established because:

(A)the number of incorrect SIDs indicated on electronic strips can be expected to be less than on paper strips, because of the possibilities of systematic checks with respect to runway allocation, runway configuration, SID allocation of the predecessor and destination in the flight plan;

(B)the allocation of an incorrect SID to an aircrew might lead to a situation in which the aircraft manoeuvres in an unanticipated way, possibly leading to a conflict with another aircraft, for example departing from a parallel runway; and

(C)an example of an acceptance criterion reads: ‘The introduction of the digital strip system should lead to a decrease of the fraction of erroneous SID allocations of more than 20 %’.

(ii)Measurability: The influence of the change on the fraction of erroneous SID allocations can be assessed before the change by means of an analysis of the causes and occurrences of such errors and the estimated efficacy of the systematic checks. The fractions can be assessed after the change by the statistics of the event reports.

(10)Finally, the last condition of independence of proxies is also satisfied. For the purpose of this example, the proxies in (5) and (6) form a set of independent proxies that are complete, i.e. they cover all identified hazards introduced by the replacement of paper strips by a digital strip system.