(286a) Fusion of EEG and Eye-Tracking Based Metrics for Characterizing the Cognitive State of Control Room Operators
AIChE Annual Meeting
Tuesday, November 15, 2022 - 8:00am to 8:25am
Traditional methods of human performance assessment ignore cognitive aspects of operatorsâ performance. On the contrary, the performance is assessed primarily based on metrics such as the number of successes/failures, deviations of operator actions from standard operating procedures, and response time (Lee et al., 2000). Such metrics fail to account for the underlying reasons (e.g., cognitive workload, shortcomings in mental models) and contributory factors (e.g., the effect of HMI design) to sub-optimal performance. Advancements in cognitive engineering and concurrent development of sensor technologies such as eye-tracking and electroencephalography (EEG) have made it possible to obtain several physiological measures that provide critical insights into human cognition and performance. Eye-tracking primarily provides information about how an operator acquires information from HMIs (reflection of operatorâs mental models) (Shahab et al., 2022) and how HMI designs influence performance (Diego-Mas et al., 2019). Likewise, electroencephalography (EEG) provides critical information about operatorsâ mental states, such as levels of cognitive workload (Liang et al., 2018). Nonetheless, eye-tracking alone provides limited insight into the mental processes; brain-sensing can complement these processes. For instance, a classification accuracy of 93% for visual hazard detection was attained when features from eye-tracking and EEG were fused compared to accuracies of 73% (for eye-tracking) and 83% (for EEG) obtained independently (Noghabaei et al., 2021). Our previous research using eye-tracking (Sharma et al., 2016, Bhavsar et al., 2017) and EEG (Iqbal et al., 2020, 2021) reveals exciting insights into the cognitive behavior of control room operators. For instance, experts have more directed attention as captured by lower gaze entropy than novices (Bhavsar et al., 2017). EEG is able to identify instances when there is a mismatch between the operatorâs mental models of process and actual process behavior (Iqbal et al., 2020).
In the current work, we study the added benefit of the fusion of eye-tracking and EEG for operator performance assessment. The proposed approach for fusion involves identifying key events during a typical period during which an operator interacts with the process. These key events could include: the periods during which an operator is simply monitoring the process when it runs normally; periods which involve disturbances; and periods during which an operator is actively taking control actions to bring the process back within normal operating conditions. The purpose of the approach is to identify which metricsâeye-tracking and/or EEG-basedâcan best capture operatorâs cognition during these events, and provide insights into their cognitive performance. For instance, whether eye-tracking based metrics are more sensitive to performance (say, level of attention) during monitoring or EEG-based. Likewise, which metrics can quickly capture increase in cognitive workload in the wake of inability of an operator to control process abnormalities, and which respond after a lag. To obtain answers to these questions, we use a simulated ethanol production process as an experimental testbed for conducting human subject studies. Ten participants were involved in the study, and each of them carried out several repetitive trials. During a typical trial, a participant has to carry out six tasks (each corresponding to a disturbance scenario). Overall, these participants performed 81 trials resulting in 438 task-participant pairs. During these tasks, participants' brain activity and eye gaze are recorded via a single electrode EEG sensor and Tobii 300 eye tracker, respectively. In addition, process data, alarm information, and operator actions are also recorded. These are used to create labels such as the monitoring phase, the period immediately after an alarm, before an operator's action, after an operator's first slider action, and the period during which the operator executes a control action. Next, eye-tracking and EEG-based features are extracted for each of these labels. These features include average pupil diameter, power spectral densities in the range of 0 to 2 Hz for pupil diameter, and power spectral densities in the range of 0-30 Hz for EEG. Decision tree-based analysis of features reveals that combining features from eye and EEG data provides more insights into operator cognitive performance than either of these. This is hypothesized due to differences in dynamics (e.g., timescale) of EEG and eye data. In this paper, we will present the proposed fusion methodology and present results from our human subject studies that demonstrate the benefits of the fusion of eye tracking and EEG based metrics.
Acknowledgement: This work is partially funded by American Express Lab for Data Analytics,
Risk and Technology, Indian Institute of Technology Madras
Bhavsar, P., Srinivasan, B., & Srinivasan, R. (2017). Quantifying situation awareness of control room operators using eye-gaze behavior. Computers & chemical engineering, 106, 191-201.
Brauer, R. L. (2016). Safety and health for engineers. John Wiley & Sons.
Diego-Mas, J. A., Garzon-Leal, D., Poveda-Bautista, R., & Alcaide-Marzal, J. (2019). User-interfaces layout optimization using eye-tracking, mouse movements and genetic algorithms. Applied ergonomics, 78, 197-209.
Iqbal, M. U., Shahab, M. A., Choudhary, M., Srinivasan, B., & Srinivasan, R. (2021). Electroencephalography (EEG) based cognitive measures for evaluating the effectiveness of operator training. Process Safety and Environmental Protection, 150, 51-67.
Iqbal, M. U., Srinivasan, B., & Srinivasan, R. (2020). Dynamic assessment of control room operatorâs cognitive workload using Electroencephalography (EEG). Computers & Chemical Engineering, 141, 106726.
Jung, S., Woo, J., & Kang, C. (2020). Analysis of severe industrial accidents caused by hazardous chemicals in South Korea from January 2008 to June 2018. Saf. Sci. 124, 104580
Lee, S., Jeong, I., & Il, M. (2000). Development of evaluation algorithms for operator training system. Computers & Chemical Engineering, 24(2-7), 1517-1522.
Liang, Y., Liang, W., Qu, J., & Yang, J. (2018, October). Experimental study on EEG with different cognitive load. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 4351-4356). IEEE.
Noghabaei, M., Han, K., & Albert, A. (2021). Feasibility Study to Identify Brain Activity and Eye-Tracking Features for Assessing Hazard Recognition Using Consumer-Grade Wearables in an Immersive Virtual Environment. Journal of Construction Engineering and Management, 147(9), 04021104.
Porthin, M., Liinasuo, M., & Kling, T. (2020). Effects of digitalization of nuclear power plant control rooms on human reliability analysisâA review. Reliability Engineering & System Safety, 194, 106415.
Shahab, M. A., Iqbal, M. U., Srinivasan, B., & Srinivasan, R. (2022). HMM-based models of control room operatorâs cognition during process abnormalities. 1. formalism and model identification. Journal of Loss Prevention in Process Industries, 104748.
Sharma, C., Bhavsar, P., Srinivasan, B., & Srinivasan, R. (2016). Eye gaze movement studies of control room operators: A novel approach to improve process safety. Computers & Chemical Engineering, 85, 43-57.