EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION EUROCONTROL EUROCONTROL EXPERIMENTAL CENTRE

Transcription

1 EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION EUROCONTROL EUROCONTROL EXPERIMENTAL CENTRE WHEELIE ADVANCED DISPLAY FILTERING TECHNIQUE (HUMAN FACTOR EXPERIMENTS) EEC Note No. 02/2008 Project: LTI-3-20-WHEE Issued: April 2008 European Organisation for the Safety of Air Navigation EUROCONTROL 2007 This document is published by EUROCONTROL in the interest of the exchange of information. It may be copied in whole or in part providing that the copyright notice and disclaimer are included. The information contained in this document may not be modified without prior written permission from EUROCONTROL. EUROCONTROL makes no warranty, either implied or express, for the information contained in this document, neither does it assume any legal liability or responsibility for the accuracy, completeness or usefulness of this information.

2 This page intentionally left blank

3 REPORT DOCUMENTATION PAGE Reference EEC Note No. 02/2008 Security Classification Unclassified Originator: EEC - LTI (Long Term Investigations) Sponsor EUROCONTROL Experimental Centre TITLE : Originator (Corporate Author) Name/Location: EUROCONTROL Experimental Centre Centre de Bois des Bordes B.P.15 F Brétigny-sur-Orge CEDEX FRANCE Telephone: +33 (0) Internet : Sponsor (Contract Authority) Name/Location (EEC) EUROCONTROL Experimental Centre Centre de Bois des Bordes B.P.15 F Brétigny-sur-Orge CEDEX FRANCE Telephone: +33 (0) Internet : WHEELIE ADVANCED DISPLAY FILTERING TECHNIQUE (HUMAN FACTOR EXPERIMENTS) Author Simone Sporer-(FH-Joaneum, University of Applied Sience, Graz) Date 03/2008 Pages xii - 67 Figures 28 Tables 46 Annexes 1 References 11 Project LTI-3-20-WHEE Task no. sponsor - Period Distribution Statement: (a) Controlled by: Marc Brochard, Head of LTI (b) Special Limitations : public (c) Copy to NTIS Descriptors (keywords) : ODS, aircraft separation, reduced complexity, mobile filtering, radar display, mouse wheel, flight level, HF, human factor experiments Abstract : This human factors study was conducted to initial investigate a novel concept idea to support air traffic controllers. The mobile horizontal filter tool WHEELIE was developed by EUROCONTROL Experimental Centre and is based on the idea that in case of vertical separation it is a priori important for the controllers to know if the aircraft are flying on the same or on different FL. WHEELIE provides the opportunity to select the requested FL by scrolling the ODS-mouse wheel and change it quickly also by activating the mouse wheel (, 180, 190, 200, ). Aircraft on the selected FL will be graphically highlighted without masking other traffic. Hypothetically WHEELIE may influence mental workload and situation awareness of controllers positively and therewith enhance safety. The project is split in two parts, the development of the simulation environment, e.g. definition of a control sector, of guiding lines and scenarios for the simulation session for the human factors study and the implementation of this study. In this report the main focus is concentrated on the conducted human factors study. The empirical study is based on a multi-level measurement approach which considers physiological variables, subjective ratings and behavioural indicators such as performance. The experimentation revealed reduced perceived mental workload and improved perceived situation awareness as a consequence of the WHEELIE tool. WHEELIE was positively assessed; especially the rapid perception of conflicts and a better overview of respective FL and their positive impact on safety were pointed out by the controllers. Ambiguous results and interfering effects as affected from the design are discussed at the end of the report.

4 This page intentionally left blank

5 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL FOREWORD Air Traffic Control (ATC) guidance through human operators rely on a three Dimensional (3D) mental traffic picture. The currently technical support is poorly adapted, as it projects the 3D traffic graphically into a 2D horizontal pane only. The third dimension - the flight level - is represented as numerical value in the aircraft label. Mental effort is required for reading, memorising and to put in relation these numeric flight levels for the creation of the human minded 3D traffic picture. In 2004, the EUROCONTROL Experimental Centre (EEC) presented the WHEELIE idea ( Wheelie A Mobile Horizontal Display Filter to ease Controller s Separation Task, proceedings of the Digital Avionic System Conference 2004, Salt Lake City, USA) to support the ATC operator. For initial investigation on the WHEELIE idea, the EEC launched a human factor study to evaluate the hypothetical benefit. The study was done in collaboration with the FH JOANNEUM, University of Applied Sciences in Graz, Austria and BARCO Orthogon, Bremen, Germany. The study is split in two parts, the design of the simulation environment and the human factor study itself. The EEC note WHEELIE Advanced Display Filtering Technique (Design of a Simulation Setup) described the development of the simulation facility. The graphical user interface for the simulation facilities was build with the ODS Toolbox (Operational Display System) of BARCO Orthogon, who provided the required software modification into ODS Toolbox to simulate the WHEELIE functionality. Further for free, they provided the WHEELIE partners with their traffic generator to animate and demonstrate the ODS Toolbox. Based on these software tools the simulation environment was build by the FH JOANNEUM. The simulation environment consists of one controller working position and a pseudo pilot position. With reference to the further human factor evaluation of WHEELIE the university developed and validated a set of operational simulation scenarios described in the EEC note. This note describes the second part of the WHEELIE project, the human factor evaluation of WHEELIE. Therefore experiments with twelfe active air traffic controll operators s from different Austrian control centres have been executed and evaluated. Horst Hering Long Term Investigation Program LTI-3-20-WHEE EEC Note No. 02/2008 v

6 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) This page intentionally left blank vi LTI-3-20-WHEE EEC Note No. 02/2008

7 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL TABLE OF CONTENTS FOREWORD... V LIST OF FIGURES... VIII LIST OF TABLES... IX 1. INTRODUCTION AND OUTLINE BACKGROUND FUNCTIONALITY OF WHEELIE STRUCTURE OF THE PROJECT STRUCTURE OF THE REPORT EVALUATION OBJECTIVES METHOD EXPERIMENTAL DESIGN Experimental Factors Measurements MATERIALS Simulation Environment Scenarios Questionnaires Psychophysiological Assessment Performance Assessment Interview Guidelines PROCEDURE PARTICIPANTS STATISTICAL ANALYSIS AND HYPOTHESIS TESTING RESULTS QUESTIONNAIRES Instantaneous Self-Assessment (ISA) NASA Task Load Index (NASA-TLX) Post-scenario Questionnaire PSYCHOPHYSIOLOGICAL ASSESSMENT Heart Rate and Heart Rate Variability PERFORMANCE ASSESSMENT INTERVIEWS Post-scenario Interview General De-briefing Interview DISCUSSION QUESTIONNAIRE RESULTS PSYCHOPHYSIOLOGICAL DATA PERFORMANCE DATA INTERVIEW RESULTS CONCLUSIONS...42 LTI-3-20-WHEE EEC Note No. 02/2008 vii

8 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) 7. REFERENCES APPENDIX...45 LIST OF ANNEXES A DESCRIPTIVE STATISTICS...46 LIST OF FIGURES Figure 1: Standard vs. WHEELIE on operational display... 1 Figure 2: Simplified control sector with restricted area... 9 Figure 3: Guiding lines for the simulation environment Figure 4: Simulation study workstation Figure 5: Average ratings of workload in WHEELIE condition and task load levels Figure 6: Average ratings of workload in task load levels Figure 7: Average ratings of perceived situation awareness in WHEELIE condition Figure 8: Average HR in bpm in 2.75min periods in task load levels Figure 9: Average HRV in ms in 5.5min periods in task load levels Figure 10: Average number of separation losses in task load levels Figure 11: Statements on the topic trigger for the usage of WHEELIE functionality Figure 12: Statement on the topic general assessment of WHEELIE functionality Figure 13: Statement on the topic advantages of WHEELIE functionality Figure 14: Statement on the topic disadvantages of WHEELIE functionality Figure 15: Statement on the topic suggestions for improvement of WHEELIE functionality Figure 16: Statement on the topic impact of WHEELIE functionality on safety and statement... on the topic improvement of detection of conflicts Figure 17: Statement on the topic impact of WHEELIE functionality on efficiency and statement on the topic impact of WHEELIE functionality on workload Figure 18: Statement on the topic impact of WHEELIE functionality on overview of traffic Figure 19: Statement on the topic implementation of WHEELIE functionality in real operation Figure 20: Average ratings of perceived physical demand in run Figure 21: Average ratings of perceived safety in scenario Figure 22: Average ratings of perceived mental workload in run Figure 23: Average HR in bpm in 2.75min periods in task load levels in run Figure 24: Average exceedances of ideal flight time in run Figure 25: Average number of wrong entry points in run Figure 26: Average number of separation losses in task load levels in run Figure 27: Number of statements on efficiency in scenario Figure 28: Number of statements on safety in run viii LTI-3-20-WHEE EEC Note No. 02/2008

9 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL LIST OF TABLES Table 1: Experimental design with factors Software, Scenario crossed... 5 Table 2: Definition of experimental groups... 5 Table 3: Experimental procedure Table 4: Results of Analysis of Variance for Instantaneous Self Assessment Table 5: Results of Analysis of Variance for NASA-TLX Table 6: Results of Analysis of Variance for post-scenario questionnaire Table 7: Results of Analysis of Variance for heart rate and heart rate variability Table 8: Results of t-test for performance assessment Table 9: Results of Analysis of Variance for performance assessment Table 10: Results of Wilcoxon test for post-scenario interview Table 11: Results of Analysis of Variance for ISA for scenario and run Table 12: Results of Analysis of Variance for NASA-TLX for scenario and run Table 13: Results of Analysis of Variance for post-scenario questionnaire for scenario and run. 33 Table 14: Results of Analysis of Variance for heart rate and heart rate variability for... scenario and run Table 15: Results of t-test for performance assessment for scenario and run Table 16: Results of Analysis of Variance for performance assessment for scenario and run Table 17: Results of Wilcoxon test for post-scenario interview for scenario and run Table 18: Descriptive statistics for Instantaneous Self Assessment Table 19: Post-test for task load (Instantaneous Self Assessment) Table 20: Descriptive statistics for Instantaneous Self Assessment for scenario Table 21: Descriptive statistics for Instantaneous Self Assessment for run Table 22: Descriptive statistics for NASA Task Load Index Table 23: Descriptive statistics for NASA Task Load Index for scenario Table 24: Descriptive statistics for NASA Task Load Index for run Table 25: Descriptive statistics for post-scenario questionnaire Table 26: Descriptive statistics for post-scenario questionnaire for scenario Table 27: Descriptive statistics for post-scenario questionnaire for run Table 28: Descriptive statistics for heart rate (in bpm) Table 29: Post-test for task load (HR) Table 30: Descriptive statistics for heart rate (in bpm) for scenario Table 31: Descriptive statistics for heart rate (in bpm) for run Table 32: Descriptive statistics for heart rate variability (in ms) Table 33: Post-test for task load (HRV) Table 34: Descriptive statistics for heart rate variability (in ms) for scenario Table 35: Descriptive statistics for heart rate variability (in ms) for run LTI-3-20-WHEE EEC Note No. 02/2008 ix

10 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) Table 36: Descriptive statistics for performance assessment Table 37: Descriptive statistics for separation losses as a function of task load levels Table 38: Post-test for task load (separation losses) Table 39: Descriptive statistics for performance assessment for scenario Table 40: Descriptive statistics for separation losses as a function of task load... levels for scenario Table 41: Descriptive statistics for performance assessment for run Table 42: Descriptive statistics for separation losses as a function of task load levels for run Table 43: Results of content analysis for post-scenario interview Table 44: Results of content analysis for post-scenario interview for scenario Table 45: Results of content analysis for post-scenario interview for run Table 46: Results of content analysis for general de-briefing interview x LTI-3-20-WHEE EEC Note No. 02/2008

11 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL ABBREVIATIONS AND ACRONYMS ANOVA ATC ATCO bpm COP df ECG F FIR FL FPS ft HR HRV IFR kts M N NATS NM ODS p RVSM SA SD SDNN t Analysis of Variance Air Traffic Control Air Traffic Controller beats per minute Coordination Point degree of freedom Electrocardiogram Fisher s F ratio Flight Information Region Flight Level Flight Progress Strip feet Heart Rate Heart Rate Variability Instrument Flight Rules knots Mean (arithmetic average) Number (total number in a sample) National Air Traffic Services Nautical Miles Operational Display Systems probability Reduced Vertical Separation Minimum Situation Awareness Standard Deviation Standard Deviation of the Normal-to-Normal Interval computed value of t test LTI-3-20-WHEE EEC Note No. 02/2008 xi

12 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) This page intentionally left blank xii LTI-3-20-WHEE EEC Note No. 02/2008

13 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL 1. INTRODUCTION AND OUTLINE 1.1. BACKGROUND Air traffic controllers (ATCO) task is to guarantee the safe separation of aircraft in the controlled airspace. Safe separation may be done in horizontal or vertical direction. Experienced controllers recognise horizontal separation at a glance on conventional two-dimensional radar displays Operational Display Systems (ODS). Vertical separation is based on flight level (FL) data which are represented in numerical mode on the ODS because the 3D traffic is graphically projected into a 2D horizontal pane. Vertical separation requires mental effort of the controllers by permanent scanning the actual FL values on the ODS, memorizing them and create therewith a mental representation of the traffic. The novel concept idea of WHEELIE to support air traffic controllers was developed by EUROCONTROL Experimental Centre (Brétigny sur Orge, France) and presented in 2004 at the Digital Avionic System Conference in Salt Lake City (Hering, 2004) FUNCTIONALITY OF WHEELIE The concept of WHEELIE is based on the idea that in case of vertical separation it is a priori more important for the controllers to know if the aircraft are flying on the same or on different FLs then to extract the exact FL numbers from the labels and compare them. The mobile horizontal filter tool WHEELIE enables the controllers to quickly answer the question if the aircraft are on the same or different FLs (Hering, 2004). The tool provides the opportunity to select the requested FL by scrolling the ODS-mouse wheel and change it quickly also by activating the mouse wheel (, 180, 190, 200, ), therefore the tool is called WHEELIE. Aircraft on the selected FL will be graphically highlighted without masking other traffic (Figure 1) and the FL is displayed on the screen in an unobtrusive way. At first glance the controllers can identify potentially conflicting aircraft on the horizontal level flying on the reference FL. All other aircraft are flying on different FLs and are out of conflict with the aircraft selected by WHEELIE. WHEELIE is designed for permanent dynamic use. Hypothetically WHEELIE may influence mental workload and situation awareness of controllers positively and therewith enhance safety. Figure 1: Standard vs. WHEELIE on operational display LTI-3-20-WHEE EEC Note No. 02/2008 1

14 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) 1.3. STRUCTURE OF THE PROJECT In the context of the project the WHEELIE software functionality is evaluated under human factor aspects. The human factors study achieves initial results concerning mental workload of the controllers as well as the acceptance of the WHEELIE software tool and safety at air traffic control (ATC). The project is split in two parts, the development of the simulation environment, e.g. definition of a control sector, of guiding lines and scenarios for the simulation session (Sporer et al, 2006), and the human factor study itself. The development of the simulation environment is documented in detail in Report D1: Definition of the demonstrator and Report D2: Development of the demonstrator. The preparation of the human factors experiments including the experimental plan for the pre-evaluation of the WHEELIE functionality is documented in Report D STRUCTURE OF THE REPORT The purpose of this document is to describe the implementation of the WHEELIE human factors study as well as the presentation and the discussion of the results. Further this final report shall sum up the whole project WHEELIE. Therefore parts of the content of the previous work packages will be attached to get a holistic overview. This document contains seven sections and one appendix which include the following information: Section 1 presents a brief introduction on the background of the project, a description of the WHEELIE functionality, and an overview of the structure of the whole project. Section 2 contains the aim and objectives of the WHEELIE evaluation. The method, including experimental design, material, procedure, and sample specification, is described in section 3. Section 4 presents the results of questionnaires, psychophysiological assessment, performance data, and interview data. The discussion of the results and the description of additional analyses concerning interfering effects are contained in section 5. Section 6 comprises the conclusion of the conducted study. Following the references (section 7), the appendix provides information on descriptive data of the statistical analyses. 2 LTI-3-20-WHEE EEC Note No. 02/2008

15 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL 2. EVALUATION OBJECTIVES The general aim of the human factors evaluation of WHEELIE is to assess the impact of the WHEELIE functionality on controller workload therewith safety and efficiency as well as on situation awareness. The specific objectives selected were: Objective 1: Evaluate whether the use of the WHEELIE functionality and the combination of the software condition and task load levels affect mental workload. Within objective 1, the following questions will be investigated: 1. Is there any significant difference in subjectively experienced mental workload between the software condition with WHEELIE functionality and without the WHEELIE functionality or between the combination of software conditions and task load levels? 2. Is there any significant difference in objective mental workload between the software condition with WHEELIE functionality and without WHEELIE functionality or between the combination of software conditions and task load levels? Objective 2: Evaluate the impact of the WHEELIE functionality and the combination of the software condition and task load levels on safety. Within objective 2, the following questions will be investigated: 1. Is there any difference in the number of separation losses between the software condition with WHEELIE functionality and without WHEELIE functionality or between the combination of software condition and task load levels? 2. Is there any difference in the number of wrong entry and exit conditions between the software condition with WHEELIE functionality and without WHEELIE functionality? 3. Is there any difference in the number of aircraft entries in the restricted area between the software condition with WHEELIE functionality and without WHEELIE functionality? 4. Is there any difference in the experienced safety level between the software condition with WHEELIE functionality and without WHEELIE functionality? Objective 3: Evaluate the impact of the WHEELIE functionality and the combination of the software condition and task load levels on efficiency. Within objective 3, the following questions will be investigated: 1. Is there any difference in the number of aircraft delays between the different software conditions with WHEELIE functionality and without WHEELIE functionality? LTI-3-20-WHEE EEC Note No. 02/2008 3

16 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) 2. Is there any difference in the experienced efficiency level between the different software conditions with WHEELIE functionality and without WHEELIE functionality? Objective 4: Evaluate whether the use of the WHEELIE functionality affects situation awareness. Within objective 4, the following questions will be investigated: 1. Is there any difference in the experienced situation awareness between the different software conditions with WHEELIE functionality and without WHEELIE functionality? Objective 5: Evaluate the acceptance of the WHEELIE functionality. Within objective 5, the following questions will be investigated: 1. Which advantages and disadvantages of the WHEELIE functionality do controllers see? 2. Which impact of the WHEELIE functionality do controllers see with respect to safety, efficiency, situation awareness, and mental workload? 3. When is the WHEELIE functionality used? Objective 6: Evaluate whether the use of the WHEELIE functionality affects emotional load. Within objective 6, the following questions will be investigated: 1. Is there any difference in experienced emotional load between the software condition with WHEELIE functionality and without WHEELIE functionality? 4 LTI-3-20-WHEE EEC Note No. 02/2008

17 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL 3. METHOD 3.1. EXPERIMENTAL DESIGN The experimental design which has been chosen for the realization of the experiment consists of three independent variables, resulting in a 2 x 2 x 5 (software condition x scenario x task load) within-subject design (table 1). The factors software condition and scenario are independently manipulated (independent permutation of each factor). Note that task load levels occur in each scenario and are not indicated separately in the design. Table 1: Experimental design with factors Software, Scenario crossed run / group software x scenario (including task load level changes) group 1 group 2 group 3 group 4 run 1 W1 W2 S1 S2 run 2 S2 S1 W2 W1 Note. W = Wheelie Condition, S = Standard Condition; 1 = Scenario 1, 2 = Scenario 2; Every participant had to perform two settings/runs one setting includes the WHEELIE condition the other one the standard condition without the WHEELIE functionality, also one setting includes scenario 1 the other one scenario 2. In each setting all levels of task load are presented in ascending ranks. The sequence of the software condition is permutated (WHEELIE Standard vs. Standard WHEELIE) as well as the sequence of the scenario (scenario1 scenario2 vs. scenario2 scenario1; permutation occurs not in combination of software and scenario). When these 4 sequences are linked together 4 groups will result as presented in table 2. Table 2: Definition of experimental groups Group1 Group2 Group3 Group4 run 1: WHEELIE condition + scenario 1 + task load ascending run 2: Standard condition + scenario 2 + task load ascending run 1: WHEELIE condition + scenario 2 + task load ascending run 2: Standard condition + scenario 1 + task load ascending run 1: Standard condition + scenario 1 + task load ascending run 2: WHEELIE condition + scenario 2 + task load ascending run 1: Standard condition + scenario 2 + task load ascending run 2: WHEELIE condition + scenario 1 + task load ascending LTI-3-20-WHEE EEC Note No. 02/2008 5

18 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) The permutation of the experimental settings at the factor software condition makes sure that potential effects of fatigue or practice will be excluded. To avoid confounded effects resulting from software condition and scenario the scenarios were also permutated. The repeated measure design benefits from the fact that possible between-group effects in consequence of e.g. age or experience will be avoided. The participants were randomly allocated to the experimental groups Experimental Factors The following experimental factors (independent variables) are manipulated in the evaluation study: 1. Software Condition, 2. Task load, and 3. Scenario. Software Condition The main manipulation in the experiment concerns the different software conditions. Two different software conditions are used: Software with WHEELIE functionality WHEELIE condition Software without WHEELIE functionality Standard condition In the software used for the experiment (ODS Toolbox provided by BARCO Orthogon GmbH) the WHEELIE functionality can be pre-selected for each experimental session. In more detail the software is described in report D1: Definition of the demonstrator and D2: Development of the demonstrator. Task load The task load within the scenarios is manipulated and can be divided in five levels: Task load demands 80% of ATCO capacity Task load demands 90% of ATCO capacity Task load demands 100% of ATCO capacity Task load demands 110% of ATCO capacity Task load demands 120% of ATCO capacity In ascending order the task load was presented in the experimental scenarios within a period of 5.5 minutes. The scenarios start with a task load level which demands 80% of ATCO capacity and end with task load demands of 120% of ATCO capacity. The different levels of task load are realized by a changing frequency of traffic. After three aircraft of starting traffic the frequency has been initially set to one aircraft each 50 seconds at step one (80% of capacity of an Controller), one aircraft each 45 seconds at step two (90%), one aircraft each 35 seconds at step three (100%), one aircraft each 25 seconds at step four (110%) and the additionally activated danger area at step five (120%). 6 LTI-3-20-WHEE EEC Note No. 02/2008

19 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL Scenario Two different experimental scenarios have been developed: Scenario 1 Scenario 2 Different scenarios have been developed that repeated measures can be applied and effects of remembering details will be eliminated. The two scenarios were created in a way that they are comparable. A preliminary study has shown that there are no differences in the difficulty of the scenarios (a full-fledged ATCO from Austro Control evaluated the scenarios concerning difficulty and comparability in general). A description of the scenarios in more detail is given in section Measurements The assessment of effects of the WHEELIE functionality on controllers is based on the multi-level measurement approach which considers physiological variables, subjective ratings and behavioural indicators such as performance. Mental workload Mental workload was measured on the basis of objective and subjective, i.e. self-assessment, measures. The following objective measurements were used: Average heart rate (HR) in bpm in 2.75-minute-intervals (10 sections of measurement) during experimental simulation sessions Average heart rate variability (HRV) in ms in 5.5-minute-intervals (5 sections of measurement) during experimental simulation sessions The following subjective measurements for mental workload were used: NASA Task Load Index (NASA-TLX), to be measured at the end of each experimental scenario Instantaneous Self-Assessment Technique (ISA), to be measured continuously during each experimental scenario (5 measurements) experienced mental workload level for the experimental scenarios, to be measured at the end of each experimental scenario (item in the post-scenario interview, item in the postscenario questionnaire) Emotional Load Emotional load was measured on the basis of subjective measurements: questionnaire and interview item on the perceived level of emotional load for the experimental scenarios (item in the post-scenario interview, item in the post-scenario questionnaire) LTI-3-20-WHEE EEC Note No. 02/2008 7

20 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) Safety The safety aspects were measured on the basis of: the number of separation losses during the experimental scenarios, the number of aircraft entries in the activated restricted area, the number of wrong exit and entry conditions The measurement on safety is obtained on the basis of the recorded flight movements during the experimental scenarios which were provided in an ASCII-file. experienced safety level for the experimental scenarios (item in the post-scenario interview, item in the post-scenario questionnaire). Efficiency Efficiency was measured on the basis of: number of aircraft delays; aircraft delays are obtained equal to the safety aspects on the basis of the recorded flight movements during the experimental scenarios. experienced efficiency level for the experimental scenarios (item in the post-scenario interview, item in the post-scenario questionnaire). Situation Awareness The aspect of Situation Awareness (SA) was measured on the basis of: perceived situation awareness for the experimental scenarios (item in the post-scenario interview, item in the post-scenario questionnaire). Acceptance The controllers acceptance of the WHEELIE functionality was measured on the basis of controllers feedback with respect to the following issues: advantages and disadvantages of the WHEELIE functionality in general impact of the tool with respect to safety, efficiency, situation awareness, and mental workload setting when the WHEELIE functionality is used. The feedback of the controllers is obtained on the basis of the general de-briefing interview. 8 LTI-3-20-WHEE EEC Note No. 02/2008

21 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL 3.2. MATERIALS Simulation Environment The simulation environment is based on the graphical user interface of the ODS Toolbox (Operational Display System) of BARCO Orthogon GmbH. They also provided the required software modification to simulate the WHEELIE functionality (option to highlight radar symbols when the specific FL is selected and display the selected FL on the screen). Moreover BARCO Orthogon GmbH added a function to record simulation sessions in a log-file (ASCII file) position data from every aircraft will be saved every five seconds. Control Sector The chosen and for the study coded control sector represents the airspace at the northern part of Austria presently used by Austro Control (called NORTH Sector). The sector has been adapted to fit to the special needs of software and scenarios. One reason for the adaptation and simplification of the control sector for the simulation study is that instructions for real Instrument Flight Rules (IFR) traffic can be very different and complex depending on special situations (e.g. direct routings instead of radar vectoring). Moreover involving all navigation points of the sector would have impacted negatively the general overview. In addition coordination points (COPs) are defined only along the flight information region (FIR) border, but for the study s needs COP s are necessary along the whole North Sector boundary to enable precisely defined coordination calls. COP s are used in this study to inform ATCOs where (in combination with a FL) a specific aircraft has to leave the controlled sector. Figure 2: Simplified control sector with restricted area LTI-3-20-WHEE EEC Note No. 02/2008 9

22 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) Additionally to the sector described above a virtual restricted area/danger area has been implemented that is always able to be activated manually, whereby vertical extensions are recommended to be from ground to FL200. If the restricted area is active a separation of at least 5NM horizontally or 500ft vertically has to be kept at all the time by vectoring aircraft around or advising them to climb above. The upper limit is chosen by 500ft like in real ATC so that the next FL above can be used without loss of separation. The implementation of this area leaves many chances to increase the difficulty of traffic samples also after hard-coding. Figure 2 demonstrates the simplified control sector including restricted area and waypoints (i.e. COPs). Guiding Lines An ATS Route (Air Traffic Services) is a specified flight route to guide the flow of air traffic on a previously defined path. For the creation of the simulation environment some routes of the global ATS network are used partly to keep a relation to real air traffic flow during the simulation. To convert flight paths into a hard-coded file the software is able to support so called guiding lines that will be followed by any aircraft having its autopilot activated. The following ATS Routes are used as guiding lines (Figure 3): Heading east: M141, M749, L605 and L725 (light-blue coloured). Heading west: L850, L851 coming to T161, L856, L610 (dark-blue coloured). Bidirectional: L604 (red coloured). Figure 3: Guiding lines for the simulation environment 10 LTI-3-20-WHEE EEC Note No. 02/2008

23 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL Pseudo Pilot and Communication The simulation environment consists of one controller working position and a pseudo pilot position. The pseudo pilot has to follow the instructions of the controllers and make the adequate input to navigate the aircrafts. In the software environment not all controller instructions are executable, e.g. direct routing instruction to any waypoint or the request climb with 1500ft per minute or more. To define accurate rules for the process of the simulation sessions to keep misleading phrases out of the samples there are only the following instructions allowed to be given by an ATCO: turn left/right heading turn left/right by degrees climb/descend to FL leave to intercept FL intercept FL reduce/increase speed to kts Instruction sheets (including also a route chart) for the controllers were placed at the working position of the controller additional to a verbal briefing on instructions allowed, software, and software related special aircraft behavior e.g. when an aircraft has been instructed to join a guiding line, it starts to turn to the direction 90 to the guiding line. Workstation The workstation consists of a controller and a pseudo pilot working position (Figure 4). Additionally there is an instructor seat behind the working places to watch the processes. The workstations of controller and pseudo pilot are divided that visual contact can not affect the study. Figure 4: Simulation study workstation LTI-3-20-WHEE EEC Note No. 02/

24 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) Flight Strips Flight Progress Strips (FPS) are used for the simulation study to define precise routing and height instructions for each flight. The FPS are generated similar to flights strips used in real ATC. These flight strips enable strip marking according standard ATC procedures. For a detailed description of the software set-up and the development of the simulation environment it is referred to report D1: Definition of the demonstrator and report D2: Development of the demonstrator Scenarios Basing on the definition of the simulation environment two training sessions and two experimental scenarios have been generated. Training scenarios The training scenarios provide the opportunity to the subjects to get to know the whole simulation environment, the control sector, guiding lines, and waypoints. Both training sessions are configured with low starting traffic to get used to the software. Thereafter continuous traffic flow starts by flights entering the control sector frequently every 45 seconds covering all available guiding lines. The scenarios are based p on the same logical structure the same number of aircraft enters the sector at exactly the same time, the same number of aircraft has to change their routing and their flight level. Except of additional effort out of unforeseeable conflict solutions both training scenarios will be comparable in their task load and difficulty. Each session will last about 15 minutes depending on ATCO conflict solution actions. The subjects start with training session 1, without the WHEELIE functionality, and after a briefing they will continue with training session 2, with active WHEELIE functionality. Experimental scenarios The structure of the two experimental scenarios is based on the same logical structure such as the training scenarios. In the experimental sessions a general proportion of 70% of all aircraft have to maintain on entry flight level unless there is conflict traffic and the residual 30% are level changes. Concerning the route changes 40% of all flights could originally remain on the entering guiding line and 60% have to change the route. To realize several steps of task load in the experimental scenarios the frequency after three aircraft of starting traffic has been initially set to one aircraft each 50 seconds at step one (80% of controller s capacity), one aircraft each 45 seconds at step two (90% of controller s capacity), one aircraft each 35 seconds at step three (100% of controller s capacity), one aircraft each 25 seconds at step four (110% of controller s capacity) and the additionally activated danger area at step five (120% of controller s capacity). The different task load levels are presented within a period of 5.5 minutes. As a result of the underlying logical structure the experimental scenarios are comparable in their difficulty and cause the same task load. They are implemented in a way that the participants of the study will not have the chance of taking any benefit (e.g. remembering details) from the previous scenario. Both experimental scenarios will last 27.5 minutes. A preliminary study has shown that there are no differences in the difficulty of the scenarios as already mentioned in section (a full-fledged ATCO from Austro Control evaluated the scenarios concerning difficulty and comparability in general). Every controller participated individually on the experimental study. 12 LTI-3-20-WHEE EEC Note No. 02/2008

25 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL Questionnaires All questionnaires except the ratings of the Instantaneous Self-Assessment Technique were presented in German language. Biographical Questionnaire A biographical questionnaire was included in the material to collect information on biographical data (age, gender, nationality) and data on training and experience (ATC training, experience as supervisor or instructor) of the participants. Initial State Questionnaire The Intitial State Questionnaire has been developed according to Janke et al (1976). It is used to better control influences that occurred before the experiment. Questions concern activities before the study, length of sleep, use of stimulants, and medication. NASA Task Load IndeX The NASA Task Load Index (TLX, Hart & Staveland, 1988) was used to investigate the subjective mental workload level at the end of each experimental scenario. The questionnaire consists of six subscales: mental demand, physical demand, performance, temporal demand, effort, and frustration. For the study a computerized version in German language (Forschungsgesellschaft für Angewandte Naturwissenschaften e.v.) has been chosen to eliminate potential effects concerning of the non native language. The German version consists exclusively of the scales. Instantaneous Self-Assessment Technique An extension of the subjective mental workload rating method to assess the mental workload during the experimental sessions is provided by the Instantaneous Self-Assessment (ISA) technique. ISA has been developed by the UK National Air Traffic Services (NATS) and provides subjective ratings of workload during the tasks on a scale of 1 (low) to 5 (high) (Stanton et al, 2005). A software tool has been developed that the controllers can assess their mental workload during the experimental sessions. LTI-3-20-WHEE EEC Note No. 02/

26 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) Post-scenario questionnaire A post-scenario questionnaire was developed for the self-reported assessment of safety, efficiency, situation awareness, mental workload, and emotional load for the experimental sessions. The questionnaire provides subjective ratings at the end of the experimental scenarios on a scale of not at all (0) to very safe, efficient (6) Psychophysiological Assessment The recording of the heart rate and heart rate variability was accomplished using Medilog AR12 digital electrocardiogram (ECG) recorder of Huntleigh Healthcare, Cardiology Products Division. A derived 12-lead ECG is provided from using five electrodes placed on the participant s chest Performance Assessment The experimental sessions are recorded in a log-file (ASCII file) where the position data from every aircraft is saved every five seconds. By means of these data separation losses (< 5nm horizontal, <1000ft vertical between two aircraft), wrong exit and entry conditions, aircraft entries in the activated danger area and aircraft delays can be analysed Interview Guidelines As well as the questionnaires the interview guidelines and the interviews were conducted in German language. Post-scenario Interview Additional to the post-scenario questionnaire a post-scenario interview was developed. The interview contains the same topics as the questionnaire (safety level, efficiency, situation awareness, mental, workload, emotional load) to collect qualitative data in addition to the quantitative. General De-briefing Interview The study was finished after a general de-briefing interview to assess the acceptance of the WHEELIE functionality. In detail the interview covers: advantages and disadvantages of the WHEELIE functionality in general, impact of the WHEELIE functionality with respect to safety, efficiency, situation awareness, and mental workload as well as the setting when the tool is used. 14 LTI-3-20-WHEE EEC Note No. 02/2008

27 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL 3.3. PROCEDURE The evaluation of the WHEELIE functionality was conducted at the FH JOANNEUM premises from February until March A handout for pre-information on the purpose and the conduction of the study was provided for the controllers. The detailed procedure of the study is demonstrated in Table 3. The evaluation experiment itself started with general information on the study and their scope. Afterwards the whole procedure, the software, and the simulation environment were described to the subjects. The biographical and Initial State Questionnaire had to be filled out after the electrodes of the ECG were attached. A 5-minute-baseline recording was taken in relaxed resting, in series 2-minute-baselines were recorded before each simulation run. The training sessions, 20 minutes each, were developed in a way that the participants had the occasion to get familiar with the software and simulation environment. During and after the training session the controllers had additionally the opportunity to ask questions concerning the study. In the experimental phase two scenarios had to be performed each with duration of 27.5 minutes. The scenario and software condition which was presented for the experimental sessions is dependent on the different experimental group membership. During the experimental scenarios the Instantaneous Self-Assessment had to be answered five times (every 5.5 minutes) on a separate notebook. The participants were instructed to accommodate the question concerning their workload as fast as possible. After finishing each experimental session the NASA-TLX and a postscenario questionnaire had to be accomplished also a post scenario interview was conducted. As closure a general de-briefing interview had to be answered by the controllers concerning the general opinion on the WHEELIE functionality. The whole study including welcome und briefing lasted approximately 3.5 hours. Table 3: Experimental procedure PROCEDURE TIME Briefing & Training 1:40 Welcome and briefing including ECG preparation 30 min Questionnaires: Biographical & Initial State Questionnaire 25 min Baseline 05 min Training scenario without WHEELIE functionality 20 min Training scenario with WHEELIE functionality 20 min Experimental Session 1:32 Baseline 02 min Experimental scenario 1 30 min Questionnaire: NASA TLX, Post-scenario questionnaire 10 min Post-scenario interview 05 min Baseline 02 min Experimental scenario 2 30 min Questionnaire: NASA TLX, Post-scenario questionnaire 10 min Post-scenario interview 05 min De-briefing 0:15 General de-briefing interview 15 min LTI-3-20-WHEE EEC Note No. 02/

28 EUROCONTROL WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) Preliminary study A preliminary study was conducted to check the whole procedure and to get familiar with it as well as to re-check the comprehensibility of the interview questions. One tower and approach controller participated at the study. As a conclusion the procedure was retained but a few changes were made to modify the interview questions for a better understandability PARTICIPANTS Participants were recruited from Austro Control at the tower and approach control in Graz and at the en-route control in Vienna. After personally contacting the tower Graz information sheets were sent to the potential participants. A sample size of n=12 was considered sufficient in this smallscale experiment. The sample group consisted of three female and nine male participants aged between 22 and 53 years (M=30.8, SD=9.4) and all of them were of Austrian nationality. Five subjects hold the controller licence for tower and approach control between 6 and 31 years (M=14.2, SD=10.6), two of the sample are on the job training for tower and approach control. The other five persons are in education and will finish the education in about 9 month and reach the controller rating STATISTICAL ANALYSIS AND HYPOTHESIS TESTING The scope of this study was to pre-evaluate the impact of the WHEELIE functionality on different dependent variables. The statistical analysis was conducted corresponding to the descriptive data analysis. The descriptive data analysis, elaborated by Abt (1987), is classified between the concept of explorative and confirmative data analysis. The descriptive hypotheses are tested on significance but do not lead to statistical decisions, contrary to confirmatory data analysis (Abt, 1987). As a result of the study design analyses of variance (ANOVA) were used for the analysis of quantitative data. Due to the independent permuted variables software condition and scenario (control variable) it is not possible to analyse all main effects on a specific indicator within one ANOVA. As the variable scenario was involved to eliminate memory effects it is not included in the main analysis. Separate analyses were conducted to investigate the effect of scenario. Software condition and task load were included in the main analysis as within-subject factors. Univariate measures were applied as they are known to have a greater psychometrical power by small-scale samples (Maxwell & Delaney, 1990). In the description of the results p-values are displayed as significant (p<.05*; p<.01**, p<.001***) or tendentially significant (p<.10(*)). Descriptive significances demonstrate the probability to falsely reject the null hypothesis even though it is true. Qualitative data were analysed with the method of content analysis by Mayring (1997). To compare the outcome of the interviews the results of the content analysis were quantified and analysed by nonparametric tests. 16 LTI-3-20-WHEE EEC Note No. 02/2008

29 WHEELIE Advanced Display Filtering Technique (Human Factor Experiments) EUROCONTROL 4. RESULTS The results of the statistical analysis are shown in the following sections. A list with mean values and standard deviations is attached in Appendix A QUESTIONNAIRES Instantaneous Self-Assessment (ISA) Subjective ratings of workload during the experimental scenario execution were presented to the controllers at 5 times at the end of each task load level i.e. within a period of 5.5 minutes. Statistical analysis included the within-subject factors software condition (with vs. without WHEELIE condition) and task load. Table 4 contains the results of the statistical analysis; corresponding descriptive statistics are presented in appendix A.1. (Table 18). A 2 x 5 repeatedmeasure ANOVA revealed a significant main effect of WHEELIE condition indicating that more mental workload arose during the simulation session without the WHEELIE functionality than with the WHEELIE functionality available (Figure 5). Also a significant main effect of task load was found (Figure 5, Figure 6). The perceived workload linearly increased from the first to the last task load level. Post-tests showed significant differences of perceived workload between the task load levels except for the last two levels 4 and 5. The results of the statistical analysis of the post-tests are attached to appendix A.1. (Table 19). No significant WHEELIE condition x task load interaction was found. Table 4: Results of Analysis of Variance for Instantaneous Self Assessment Results (F df hypothesis, df error, p-value) Source Perceived mental workload WHEELIE F 1, 11 =6.71, p=.025* TASK LOAD F 3.49, =31.91, p=.000*** a WHEELIE x TASK LOAD F 3.42, =1.16, p=.340 Note: Trend effects: a linear; ***p<.001, **p<.01, *p<.05, (*)p<.10 LTI-3-20-WHEE EEC Note No. 02/