Horizon CDT Research Highlights

Research Highlights

Real-time Mental Workload Measure and Feedback using fNIRS

  Horia Maior (2012 cohort)   www.nottingham.ac.uk/~psxhama

Abstract:

Understanding and identifying individuals’ capabilities and limitations has always been a challenge within work contexts, but its importance cannot be underestimated. Humans are known for having a limited mental capacity, which means that they can only perform a finite set of tasks at any given period of time. Identifying these limitations is a key factor in the reduction and prevention of what is referred to as Mental Workload (MWL) Overload [2]. Methods have ranged from subjective measures such as Nasa-TLX to quantitative measures utilizing a mixture of task performance and physiological measures [5]. Physiological measures are typically task independent and therefore provide a way of objectively quantifying tasks that are typically difficult to do so. These physiological measures, whether directly or not, aim to characterize the participants cognition during the task e.g. Eye tracking has been correlated to mental workload induced during user studies.

fNIRS - Source detector diagram and placement.

SAM Howard

SAM Howard

My research is focused into the integration of physiological sensors and non-invasive brain monitoring devices (such as fNIRS), in the field of HCI. More specifically, I am looking at how we can use such devices to capture physiological responses to human cognition and mental workload, and provide the user with concurrent feedback of their mental workload during tasks.

fNIRS is a brain imaging technique that offers the potential to provide continuous, detailed insight into human mental workload, enabling an objective means of detecting overload conditions during complex tasks.

PhD Aim

  • Move towards using a more quantifiable-objective measure of mental workload in the area of Human Computer Interaction (HCI) and Human Factors (HF) evaluation methodology.
  • Understand task elements inducing various levels of mental workload.
  • Using non-invasive brain sensing technology functional Near Infrared Spectroscopy (fNIRS) to distinguish in real time between high and low workload situations.
  • Using fNIRS real-time workload measure to provide real time workload feedback to the user, meant to support the user in moments of low/high workload conditions to maintain a “good” performance.
  • Understanding the impact on user's decision making of the fNIRS workload feedback element during low/high workload situations.

Research Questions

R0 - How can we use fNIRS as a useful measure of workload in realistic HCI-HF settings?

  • Investigate fNIRS feasibility of use in HCI-HF settings [1]. Understand how we can distinguish using fNIRS when operators/users are “busy“ (performing a task) and when they are under a “rest” condition (not performing any task) [1]?
  • Understand the impact on fNIRS signal of various artifacts produced in normal HCI-HF settings [1]. Investigate methods to “handle” these artifacts. Understand the impact of various artifacts on the two different task encoding: spatial task vs verbal [1].
  • Investigate the Reliability/Replicability of fNIRS measure [1].
  • Investigating the Sensitivity of the fNIRS measure: Understand the sensitivity of the measure to both spatial and verbal memory tasks [3,4]. Investigate methods to distinguish between various levels of workload using fNIRS [3,4].

R1 - How can we use these methods/measures to gain insights into mental workload during tasks that vary in complexity, and what are the properties of tasks that elicit/impact workload?

  • Understand how fNIRS measure is best used post-study analysis [1,3,4].
  • Understand how fNIRS measure is best used in Real-Time analysis [4].

R2 - What is the impact of presenting a real-time workload feedback to the user during task that vary in complexity?

  • Understanding the effect real time perceived workload has on operator’s/user’s decision making during high/low demand tasks.
  • Studying operator trust in technology (the feedback element).

References

  1. Maior, H. A. Pike, M., Wilson, M. L., and Sharples, S. (In Press) Examining the Reliability of Using fNIRS in Realistic HCI Settings for Spatial and Verbal Tasks. In: CHI'15 ACM SIGCHI Conference on Human Factors in Computer Systems, Seoul, Korea, April 2015
  2. Maior, H. A. and Pike, M. (2014). Measuring Work Overload. In The Ergonomist magazine of the Institute of Ergonomics and Human Factors (IEHF), No. 527 May 2014.
  3. Pike, M., Maior, H. A., Porcheron, M., Sharples, S. and Wilson, M. L. (2014). Measuring the effect of Think Aloud Protocols on Workload using fNIRS. In: CHI'14 ACM SIGCHI Conference on Human Factors in Computer Systems, Toronto, April-May 2014.
  4. Maior, H. A., Pike, M., Wilson, M. L., and Sharples, S. CONTINUOUS DETECTION OF WORKLOAD OVERLOAD: AN FNIRS APPROACH. In Contemporary Ergonomics and Human Factors 2014: Proceedings of the international conference on Ergonomics & Human Factors 2014, Southampton, UK, 7-10 April 2014 (p. 450).
  5. Maior, H. A. , Pike, M., Wilson, L.M. and Sharples, S. (2013) Directly Evaluating the Cognitive Impact of Search User Interfaces: a Two-Pronged Approach with fNIRs EuroHCIR 2013 Workshop, Dublin, Ireland, 1st August 2013

Publications

  1. Lukanov, K., Maior, H. and Wilson, M. L. (2016) Using fNIRS in Usability Testing: Understanding the Effect of Web Form Layout on Mental Workload. In CHI'16: 33rd Annual ACM Conference on Human Factors in Computing Systems. May 7-12, San Jose, CA, USA.
  2. Maior, H. A. Pike, M., Wilson, M. L., and Sharples, S. (In Press) Examining the Reliability of Using fNIRS in Realistic HCI Settings for Spatial and Verbal Tasks. In: CHI'15 ACM SIGCHI Conference on Human Factors in Computer Systems, Seoul, Korea, April 2015

This author is supported by the Horizon Centre for Doctoral Training at the University of Nottingham (RCUK Grant No. EP/G037574/1) and by the RCUK’s Horizon Digital Economy Research Institute (RCUK Grant No. EP/G065802/1).