ATS.OR.205 Safety assessment and assurance of changes to the functional system

GENERAL

(a)The safety assessment should be conducted by the air traffic services provider itself. It may also be carried out by another organisation, on its behalf, provided that the responsibility for the safety assessment remains with the air traffic services provider.

(b)A safety assessment needs to be performed when a change affects a part of the functional system managed by the provider of air traffic services and that is being used in the provision of its (air traffic) services. The safety assessment or the way it is conducted does not depend on whether the change is a result of a business decision or a decision to improve safety.

SCOPE OF THE CHANGE

(a)The description of the elements being changed includes the nature, functionality, location, performance, maintenance tasks, training and responsibilities of these elements, where applicable. The description of interfaces and interactions, between machines and between humans and machines, should include communication means, e.g. language, phraseology, protocol, format, order and timing and transmission means, where applicable. In addition, it includes the description of the context in which they operate.

(b)There are two main aspects to consider in evaluating the scope of a change:

(1)The interactions within the changed functional system;

(2)The interactions within the changing functional system, i.e. those that occur during transitions from the current functional system to the changed functional system. During such transitions, components are replaced/installed in the functional system. These installation activities are interactions within the changing functional system and are to be included within the scope of the change. As each transition can be treated as a change to the functional system, the identification of both the above has a common approach described below.

(c)The scope of the change is defined as the set of the changed components and affected components. In order to identify the affected components and the changed components, it is necessary to:

(1)know which components will be changed;

(2)know which component’s (components’) behaviour might be directly affected by the changed components, although it is (they are) not changed itself (themselves);

(3)detect indirectly affected components by identifying:

(i)new interactions introduced by the changed or directly affected components; and/or

(ii)interactions with changed or directly affected components via the environment.

(4)Furthermore, directly and indirectly affected components will be identified as a result of applying the above iteratively to any directly and indirectly affected components that have been identified previously. The scope of the change is the set of changed, directly impacted and indirectly impacted components identified when the iteration identifies no new components.

(d)The context in which the changed service is intended to operate (see ATS.OR.205(a)(1)(iii)) includes the interface through which the service will be delivered to its users.

TRAINING If the change modifies the way people interact with the rest of the functional system, then a training might be required before the change becomes operational. Care should be taken when training operational staff before the change is operational, as the training may change the behaviour of the operational staff when they interact with the existing functional system before any other part of the change is made, and so may have to be treated as a transitional stage of the change. For example, as a result of training, air traffic controllers (ATCOs) may come to expect information or alerts to be presented differently. People may also need refreshment training periodically in order to ensure that their performance does not degrade over time. The training needed before operation forms part of the design of the change, while the refreshment training is part of the maintenance of the functional system after the change is in operation.

DESCRIPTION OF THE SCOPE — ‘MULTI-ACTOR CHANGE’ In reference to ‘multi-actor change’, please refer to GM1 ATM/ANS.OR.C.005(b)(1) Safety support assessment and assurance of changes to the functional system.

INTERACTIONS The identification of changed interactions is necessary in order to identify the scope of the change because any changed behaviour in the system comes about via a changed interaction. Changed interaction happens via an interaction at an interface of the functional system and the context in which it operates. Consequently, identification of both interfaces and interactions is needed to be sure that all interactions have identified interfaces and all interfaces have identified interactions. From this, all interactions and interfaces that will be changed can be identified.

FORM OF ASSURANCE The air traffic services provider should ensure that the assurance required by ATS.OR.205(a)(2) is documented in a safety case.

COMPLETENESS OF THE ARGUMENT The argument should be considered complete when it shows, as applicable, that:

(a)the safety assessment in ATS.OR.205(b) has produced a sufficient set of non-contradictory valid safety criteria;

(b)safety requirements have been placed on the elements changed and on those elements affected by the change;

(c)the safety requirements as implemented meet the safety criteria;

(d)all safety requirements have been traced from the safety criteria to the level of the architecture at which they have been satisfied;

(e)each component satisfies its safety requirements;

(f)each component operates as intended, without adversely affecting the safety; and

(g)the evidence is derived from known versions of the components and the architecture and known sets of products, data and descriptions that have been used in the production or verification of those versions.

ASSURANCE — SOFTWARE

(a)When a change to a functional system includes the introduction of new software or modifications to existing software, the ATS provider should ensure the existence of documented software assurance processes necessary to produce evidence and arguments that demonstrate that the software behaves as intended (software requirements), with a level of confidence consistent with the criticality of the required application.

(b)The ATS provider should use the software experience gained to confirm that the software assurance processes are effective and, when used, the allocated software assurance levels (SWALs) and the rigour of the assurances are appropriate. For that purpose, the effects from a software malfunction (i.e. the inability of a programme to perform a required function correctly) or failure (i.e. the inability of a programme to perform a required function) reported according to the relevant requirements on reporting and assessment of service occurrences should be assessed in comparison with the effects identified for the system concerned as per the severity classification scheme.

ASSURANCE — SOFTWARE ASSURANCE PROCESSES

(a)The software assurance processes should provide evidence and arguments that they, as a minimum, demonstrate the following:

(1)The software requirements correctly state what is required by the software, in order to meet the upper level requirements, including the allocated system safety requirements as identified by the safety assessment of changes to the functional system (AMC2 ATS.OR.205(a)(2)). For that purpose, the software requirements should:

(i)be correct, complete and compliant with the upper level requirements; and

(ii)specify the functional behaviour, in nominal and downgraded modes, timing performances, capacity, accuracy, resource usage on the target hardware, robustness to abnormal operating conditions and overload tolerance, as appropriate, of the software.

(2)The traceability is addressed in respect of all software requirements as follows:

(i)Each software requirement should be traced to the same level of design at which its satisfaction is demonstrated.

(ii)Each software requirement allocated to a component should either be traced to an upper level requirement or its need should be justified and assessed that it does not affect the satisfaction of the safety requirements allocated to the component.

(3)The software implementation does not contain functions that adversely affect safety.

(4)The functional behaviour, timing performances, capacity, accuracy, resource usage on the target hardware, robustness to abnormal operating conditions and overload tolerance, of the implemented software comply with the software requirements.

(5)The software verification is correct and complete, and is performed by analysis and/or testing and/or equivalent means, as agreed with the competent authority.

(b)The evidence and arguments produced by the software assurance processes should be derived from:

(1)a known executable version of the software;

(2)a known range of configuration data; and

(3)a known set of software items and descriptions, including specifications, that have been used in the production of that version, or can be justified as applicable to that version.

(c)The software assurance processes should determine the rigour to which the evidence and arguments are produced.

(d)The software assurance processes should include the necessary activities to ensure that the software life cycle data can be shown to be under configuration control throughout the software life cycle, including the possible evolutions due to changes or problems’ corrections. They should include, as a minimum:

(1)configuration identification, traceability and status accounting activities, including archiving procedures;

(2)problem reporting, tracking and corrective actions management; and

(3)retrieval and release procedures.

(e)The software assurance processes should also cover the particularities of specific types of software such as COTS, non-development software and previously developed software where generic assurance processes cannot be applied. The software assurance processes should include other means to give sufficient confidence that the software meets the safety objectives and requirements, as identified by the safety risk assessment and mitigation processes. If sufficient assurance cannot be provided, complementary mitigation means aiming at decreasing the impact of specific failure modes of this type of software, should be applied. This may include but is not limited to:

(1)software and/or system architectural considerations;

(2)existing service level experience; and

(3)monitoring.

SAFETY CRITERIA ‘Safety criteria will remain satisfied’ means that the safety criteria continue to be satisfied after the change is implemented and put into operation. The safety case needs to provide assurance that the monitoring requirements of ATS.OR.205(b)(6) are suitable for demonstrating, during operation, that the safety criteria remain satisfied and, therefore, the argument remains valid.

ASSURANCE LEVELS The use of assurance level concepts, e.g. design assurance levels (DAL), software assurance levels (SWAL), hardware assurance levels (HWAL), can be helpful in generating an appropriate and sufficient body of evidence to help establish the required confidence in the argument.

SAFETY REQUIREMENTS The following non-exhaustive list contains examples of safety requirements that specify:

(a)for equipment, the complete behaviour, in terms of functions, accuracy, timing, order, format, capacity, resource usage, robustness to abnormal conditions, overload tolerance, availability, reliability, confidence and integrity; The complete behaviour is limited to the scope of the change. Safety requirements should only apply to the parts of a system affected by the change. In other words, if parts of a system can be isolated from each other and only some parts are affected by the change, then these are the only parts that are of concern;

(b)for people, their performance in terms of tasks (e.g. accuracy, response times, acceptable workload, reliability, confidence, skills, and knowledge in relation to their tasks);

(c)for procedures, the circumstances for their enactment, the resources needed to perform the procedure (i.e. people and equipment), the sequence of actions to be performed and the timing and accuracy of the actions; and

(d)interactions between all parts of the system.

SAFETY ASSESSMENT METHODS

(a)The air traffic services provider can use a standard safety assessment method or it can use its own safety assessment method to assist with structuring the process. However, the application of a method is not a guarantee of the quality of the results. It is therefore not sufficient for a safety case to claim that the assurance provided is adequate due to compliance with a standard or method.

(b)There are databases available that describe different safety assessment methods, tools and techniques that can be used by the air traffic services provider. The provider must ensure that the safety assessment method is adequate for the change being assessed and that the assumptions inherent in the use of the method are recognised and accommodated appropriately.

COMPLETENESS OF HAZARD IDENTIFICATION The air traffic services provider should ensure that hazard identification:

(a)targets complete coverage of any condition, event, or circumstance related to the change, which could, individually or in combination, induce a harmful effect;

(b)has been performed by personnel trained and competent for this task; and

(c)need only include hazards that are generally considered as credible.

HAZARDS TO BE IDENTIFIED The following hazards should be identified:

(a)New hazards, i.e. those introduced by the change relating to the:

(1)failure of the functional system; and

(2)normal operation of the functional system; and

(b)Already existing hazards that are affected by the change and are related to:

(1)the existing parts of the functional systems; and

(2)hazards outside the functional system, for example, those inherent to aviation.

HAZARD IDENTIFICATION

(a)Completeness of hazard identification In order to achieve completeness in the identification of hazards, it might be beneficial to aggregate hazards and to formulate them in a more abstract way, e.g. at the service level. This might in turn have drawbacks when analysing and evaluating the risk of the hazards. The appropriate level of detail in the set of hazards and their formulation, therefore, depends on the change and the way the safety assessment is executed. Only credible hazards need to be identified. A credible hazard is one that has a material effect on the risk assessment. A hazard will not be considered credible when it is either highly improbable that the hazard will occur or that the accident trajectories it initiates will materialise. In other words, a hazard need not be considered if it can be shown that it induces an insignificant risk.

(b)Sources of hazards

(1)Hazards introduced by failures or nominal operations of the ATM/ANS functional systems may include the following factors and processes:

(i)design factors, including equipment, procedural and task design;

(ii)operating practices, including the application of procedures under actual operating conditions and the unwritten ways of operating;

(iii)communications, including means, terminology, order, timing and language and including human–human, human–machine and machine–machine communications;

(iv)installation issues;

(v)equipment and infrastructure, including failures, outages, error tolerances, nuisance alerts, defect defence systems and delays; and

(vi)human performance, including restrictions due to fatigue and medical conditions, and physical limitations, when considered relevant to the change assessment.

(2)Hazards introduced in the context in which the ATM/ANS functional system operates may include the following factors and processes:

(i)wrong, insufficient or delayed information and inadequate services delivered by third parties;

(ii)personnel factors, including working conditions, company policies for and actual practice of recruitment, training and allocation of resources, when considered relevant to the change;

(iii)organisational factors, including the incompatibility of production and safety goals, the allocation of resources, operating pressures and the safety culture;

(iv)work environment factors such as ambient noise, temperature, lighting, annoyance, ergonomics and the quality of man–machine interfaces; and

(v)external threats such as fire, electromagnetic interference and sources of distraction, when considered relevant to the change.

(3)The hazards introduced in the context in which the ATM/ANS services are delivered may include the following factors and processes:

(i)errors, failures, non-compliance and misunderstandings between the airborne and ground domains;

(ii)traffic complexity, including traffic growth, fleet mix and different types of traffic, when considered relevant to the change;

(iii)wrong, insufficient or delayed information delivered by third parties;

(iv)inadequate service provisioning by third parties; and

(v)external physical factors, including terrain, weather phenomena, volcanoes and animal behaviour, when considered relevant to the change.

(c)Methods to identify hazards

(1)The air traffic services provider may use a combination of tools and techniques, including functional analysis, what if techniques, brainstorming sessions, expert judgement, literature search (including accident and incident reports), queries of accident and incident databases in order to identify hazards.

(2)The air traffic services provider needs to make sure that the method is appropriate for the change and produces (either individually or in combination) a valid (necessary and sufficient) set of hazards. This may be aided by drawing up a list of the functions associated with part of the functional system being changed. The air traffic services provider needs to make sure their personnel that use these techniques are appropriately trained to apply these methods and techniques.

DETERMINATION OF THE SAFETY CRITERIA FOR THE CHANGE When determining the safety criteria for the change being assessed, the air traffic services provider should, in accordance with ATS.OR.210, ensure that:

(a)the safety criteria support a risk analysis that is:

(1)relative or absolute, i.e. refers to:

(i)the difference in safety risk of the system due to the change (relative); or

(ii)the difference in safety risk of the system and a similar system (can be absolute or relative); and

(iii)the safety risk of the system after the change (absolute); and

(2)objective, whether risk is expressed numerically or not;

(b)the safety criteria are measurable to an adequate degree of certainty;

(c)the set of safety criteria can be represented totally by safety risks, by other measures that relate to safety risk or a mixture of safety risks and these other measures;

(d)the set of safety criteria should cover the change; the safety criteria selected are consistent with the overall safety objectives established by the air traffic services provider through its SMS and represented by its annual and business plan and safety key performance indicators; and

(e)where a safety risk or a proxy cannot be compared against its related safety criteria with acceptable certainty, the safety risk should be constrained and actions should be taken, in the long term, so as to manage safety and ensure that the air traffic services provider’s overall safety objectives are met.

COMPLETENESS OF RISK ANALYSIS The air traffic services provider should ensure that the risk analysis is carried out by personnel trained and competent to perform this task and should also ensure that:

(a)a complete list of harmful effects in relation to the identified:

(1)hazards, when the safety criteria are expressed in terms of safety risk, or proxies, when the safety criteria are expressed in relation to proxies; and

(2)hazards introduced due to implementation is produced; and

(b)the risk contributions of all hazards and proxies are evaluated; and

(c)risk analysis is conducted in terms of risk or in terms of proxies or a combination of them, using specific measurable properties that are related to operational safety risk; and

(d)results can be compared against the safety criteria.

SEVERITY CLASSIFICATION OF ACCIDENTS LEADING TO HARMFUL EFFECTS When performing a risk analysis in terms of risk, the air traffic services provider should ensure that the harmful effects of all hazards are allocated a safety severity category and that, where there is more than one safety severity category of harm, any severity classification scheme satisfies the following criteria:

(a)The scheme is independent of the causes of the accidents that it classifies, i.e. the severity of the worst accident does not depend upon whether it was caused by an equipment malfunction or human error;

(b)The scheme permits unique assignment of every harmful effect to a severity category;

(c)The severity categories are expressed in terms of a single scalar quantity and in terms relevant to the field of their application;

(d)The level of granularity (i.e. the span of the categories) is appropriate to the field of their application;

(e)The scheme is supported by rules for assigning a harmful effect unambiguously to a severity category; and

(f)The scheme is consistent with the air traffic services providers views of the severity of the harmful effects covered and can be shown to incorporate societal views of their severity.

RISK EVALUATION The air traffic services provider should ensure that the risk evaluation includes:

(a)an assessment of the identified hazards for a notified change, including possible mitigation means, in terms of risk or in terms of proxies or a combination of them;

(b)a comparison of the risk analysis results against the safety criteria taking the uncertainty of the risk assessment into account; and

(c)the identification of the need for risk mitigation or reduction in uncertainty or both.

RISK MITIGATION When the risk evaluation results show that the safety criteria cannot be satisfied, then the air traffic services provider should either abandon the change or propose additional means of mitigating the risk. If risk mitigation is proposed, then the air traffic services provider should ensure that it identifies:

(a)all of the elements of the functional system, e.g. training, procedures that need to be reconsidered; and

(b)for each part of the amended change, those parts of the safety assessment (requirements from (1) to (6) listed in ATS.OR.205(b)) that need to be repeated in order to demonstrate that the safety criteria will be satisfied.

RISK ANALYSIS IN TERMS OF SAFETY RISK

(a)Risk analysis When a risk assessment of a set of hazards is executed, in terms of risk:

(1)the frequency or probability of the occurrence of the hazard should be determined;

(2)the possible sequences of events from the occurrence of a hazardous event to the occurrence of an accident, which may be referred to as accident trajectories, should be identified. The contributing factors and circumstances that distinguish the different trajectories from one another should also be identified, as should any mitigations between a hazardous event and the associated accident;

(3)the potential harmful effects of the accident, including those resulting from a simultaneous occurrence of a combination of hazards, should be identified;

(4)the severity of these harmful effects should be assessed, using a defined severity scheme according to point (f) of AMC2 ATS.OR.205(b)(3); and

(5)the risk of the potential harmful effects of all the accidents, given the occurrence of the hazard, should be determined, taking into account the probabilities that the mitigations may fail as well as succeed, and that particular accident trajectories will be followed when particular contributing factors and circumstances occur.

(b)Severity schemes The severity determination should take place according to a severity classification scheme. The purpose of a severity classification scheme is to facilitate the management and control of risk. A severity class is, in effect, a container within which accidents can be placed if their severities are considered similar. Each container can be given a value which represents the consequences, i.e. small for accidents causing little harm and big for accidents causing a lot of harm. The sum of the probabilities of all the accidents assigned to a severity class multiplied by the value that is related to the severity class, is the risk associated with that class. If the value that represents severity for all classes is scalar, then the total risk is the sum of the risks in each severity class.

(1)Single-risk value severity schemes Such schemes use a single severity category to represent harm to humans. Other categories representing other kinds of harm e.g. damage to aircraft and loss of separation, may be present but do not represent harm to humans. In these circumstances, risk analysis would actually be reduced to frequency/probability analysis.

(2)Multiple-risk value severity schemes Multiple-risk value severity schemes, which use a number of severity categories to classify different levels of harm, facilitate the management and control of risk in a number of ways. At the simplest level, the distribution of accidents across the severity classes gives a picture of whether the risk profile of a system is well balanced. For example, many accidents in the top and bottom severity classes with few in between suggests an imbalance in risk, perhaps due to an undue amount of attention having been paid to some types of accident at the expense of others. More detailed management and control of risk includes:

(i)Severity classes may be used as the basis for reporting accident statistics.

(ii)Severity classes combined with frequency (or probability) classes can be used to define criteria for decision-making regarding risk acceptance.

(iii)The total risk associated with one or more severity classes can be managed and controlled. For example, the sum of the risk from all severity classes represents the total risk and may be used as a basis for making decisions about changes.

(iv)Similarly, the risk associated with accident types of different levels of severity can be compared. For example, comparing runway infringement accidents with low speed taxiway accidents would allow an organisation to focus their efforts on mitigating the accident type with greatest risk.

(c)The air traffic services provider should coordinate its severity scheme(s) when performing multi-actor changes to ensure adequate assessment. This includes coordination with air traffic services providers outside of the EU.

VERIFICATION The air traffic services provider should ensure that verification activities of the safety assessment process include verification that:

(a)the full scope of the change is addressed throughout the whole assessment process, i.e. all the elements of the functional system or environment of operation that are changed and those unchanged elements that depend upon them and on which they depend are identified;

(b)the way the service behaves complies with and does not contradict any applicable requirements placed on the changed service or the conditions attached to the providers certificate;

(c)the specification of the way the service behaves is complete and correct;

(d)the specification of the operational context is complete and correct;

(e)the risk analysis is complete as per AMC1 ATS.OR.205(b)(3);

(f)the safety requirements are correct and commensurate with the risk analysis;

(g)the design is complete and correct with reference to the specification and correctly addresses the safety requirements;

(h)the design was the one analysed; and

(i)the implementation, to the intended degree of confidence, corresponds to that design and behaves only as specified in the given operational context.

OUTCOME OF RISK EVALUATION The purpose of risk evaluation is to evaluate the risk of the change and to compare that against the safety criteria with the following outcomes in mind:

(a)A possible (desired) outcome is that the assessed risk satisfies the safety criteria. This implies that the change is assessed as sufficiently safe to implement.

(b)Another possible outcome is that the assessed risk does not satisfy the safety criteria. This might lead to the decision to refine the risk analysis, to the decision to add mitigating means, or to the decision to abandon the change.

RISK EVALUATION — UNCERTAINTY

(a)The outcome of a risk analysis is uncertain due to modelling, estimates, exclusion of rare circumstances or contributing factors, incident and safety event underreporting, false or unclear evidence, different expert opinions, etc. The uncertainty may be indicated explicitly, e.g. by means of an uncertainty interval, or implicitly, e.g. by means of a reference to the sources the estimates are based upon.

(b)Where possible sequences of events, contributing factors and circumstances are excluded in order to simplify the risk estimate, which may be necessary to make the estimate of risks feasible, arguments and evidence justifying this should be provided in the safety case. This may result in increasing the uncertainty of the risk estimations.

RISK EVALUATION — FORMS OF RISK EVALUATION The risk evaluation can take several forms, even within the safety assessment of a single change, depending on the nature of the risk analysis and the safety criteria:

(a)If a set of safety requirements has been created and can be unambiguously and directly related to the safety criteria, then the risk evaluation takes the form of justifying that these requirements satisfy the safety criteria;

(b)If the safety criteria have been established in terms of the likelihood of the hazards and the severity of their effects, then the risk evaluation takes the form of verifying that the assessed risks satisfy the safety criteria in terms of risks; and

(c)If the values of all relevant proxies have been determined, then the risk evaluation takes the form of verifying that these values satisfy the safety criteria in terms of proxies.

TYPE OF RISK MITIGATION Risk mitigation may be achieved in the following ways:

(a)an improvement of the performance of a functional subsystem;

(b)an additional change of the ATM/ANS functional system;

(c)an improvement of the services delivered by third parties;

(d)a change in the physical environment; or

(e)any combination of the above-mentioned methods.

VERIFICATION OF SAFETY CRITERIA As the complete behaviour of the change is reflected in satisfying the safety criteria for the change, no safety requirements are set at system or change level. Nevertheless, safety requirements can be placed on the architecture and the components affected by the change.

MONITORING OF INTRODUCED CHANGE The air traffic services provider should ensure that within the safety assessment process for a change, the monitoring criteria, that are to be used to demonstrate that the safety case remains valid during the operation of the changed functional system, are identified and documented. These criteria are specific to the change and should be such that they indicate that:

(a)the assumptions made in the argument remain valid;

(b)critical proxies remain as predicted in the safety case and are no more uncertain; and

(c)other properties that may be affected by the change remain within the bounds predicted by the safety case.

MONITORING OF INTRODUCED CHANGE

(a)Monitoring is intended to maintain confidence in the safety case during operation of the changed functional system. At entry into service, the safety criteria become performance criteria rather than design criteria. Monitoring is, therefore, only applicable following entry into service of the change.

(b)Monitoring is likely to be of internal parameters of the functional system that provide a good indication of the performance of the service. These parameters may not be directly observable at the service level, i.e. at the interface of the service with the operational context. For example, where a function is provided by multiple redundant resources, the availability of the function will be so high that monitoring it may not be useful. However, monitoring the availability of individual resources, which fail much more often, may be a useful indicator of the performance of the overall function.