Data-Driven Assessment of RAF Trainee Performance using Eye-Tracking in Varjo Mixed-Reality Headsets
Pilot training within defence is a long, complex and resource-intensive process.
Military pilot training in the UK is conducted within the UK Military Flying Training System (UKMFTS): a collaborative partnership between the UK Armed Forces, National Armaments Director Group and Industry, led by Ascent Flight Training. With training spanning several years, it is a priority to avoid attrition in the later stages, when investment in a trainee is already significant.
Traditional assessment methods struggle to objectively measure the human factors that underpin effective airmanship, making it difficult to identify issues early and intervene in a targeted way.
Cineon’s TACET-M can help address this challenge, by enabling a Mixed Reality training and assessment capability that provides objective, data-driven insight into pilot airmanship. By combining eye-tracking, AI and Mixed Reality, TACET-M enables instructors to understand how pilots perceive, prioritise and respond to complex scenarios, supporting earlier and more effective training interventions and helping to reduce costly late-stage failures.
The Technology
Mixed Reality headsets such as the Varjo XR4 Series integrated with fixed-base simulators deliver an effective platform for training and assessment, that also reduce bottlenecks associated with aircraft availability. Critically, they can also provide a rich source of additional data that can used to create more objective and tailored training interventions.
One source of data is from biosensors built into the headsets as standard. There is now a compelling and concrete evidence-base for the use of eye-movement data and data from other sensors to understand performance in terms of cognitive and psychological factors (including attention, stress, cognitive load and fatigue).
The effects of cognitive state on visual attention have been well-studied by psychological researchers. Where we direct our visual attention is guided by both our current goals and motivations (topdown) and visual characteristics of the environment (bottom-up). Different brain networks underlay these attentional systems. These networks work cooperatively under normal conditions. However, either fatigue or highly
demanding or stressful situations can affect attentional control, reducing the effectiveness of top-down (goals and motivations) processing. This leads to an over-reliance on bottom-up (stimulus driven) visual attention. This switch in attentional control affects how the eyes behave. Studies based on experiments that require some inhibition (or target-locking) of gaze have consistently provided evidence for reduced top-down control of eye movements under stress, workload, and fatigue.
Eye-tracking can also be used to explore patterns and efficiencies in eye-movements that are associated with expertise. These factors can be linked to core ‘airmanship’ principles as part of a competency-based assessment. The psychology literature identifies a concrete link between eye-movement data and airmanship. For example, Endsley’s Theory of Situational Awareness (SA) identifies Level 1 SA i.e. the perception of cues, as relying on effective visual scanning to pick-up relevant information in order to build an understanding & comprehension of the environment, which constitutes Level 2 SA. The same link plays a part in a person’s ability to forecast future events i.e. Level 3 SA.
While these links have been discussed in the literature for some years, to be effective these signals require complex pre-processing; a step that has been aided by advances in machine learning. This has enabled the creation of reliable construct models to underpin performance assessments. It can also allow for the inclusion of other potentially conflicting signals such as pupil size and heartrate, which are sometimes inaccurately cited as direct proxies for stress and cognitive load.
The Challenge
Defence pilot training organisations must balance operational demand with limited training resources, high simulator costs and long training timelines. While technical flying skills can be measured with relative ease, airmanship qualities remain difficult to assess objectively, for example:
- Situational Awareness (inc. Positional Situational Awareness and Tactical Situational Awareness)
- Mental Capacity (inc. Situational Analysis, Priority Allocation and Mental Flexibility)
- Decisiveness (inc. Decision Making, Decision Quality and Decision Implementation)
- Communications (inc. Internal and External Communications)
- Resource Management (inc. Systems Management, Cockpit Management, Internal and External Resource Management)
Key challenges include:
- Late‑stage attrition, where trainees fail after years of training investment
- Limited insight into human factors, with assessments often reliant on subjective judgement
- High dependence on full‑flight simulators, increasing cost and reducing availability
- Difficulty replicating complex, high‑stress or ‘black swan’ scenarios in a safe and repeatable way
These limitations make it harder to identify trainees who are struggling early in the pipeline and to deliver targeted interventions that could keep them on track.
How Mixed Reality, Eye-tracking and AI Enable Objective Airmanship Assessment
To overcome these challenges, Cineon developed TACET-M as a defence-focused extension of its flagship TACET platform. The solution introduces objective, data-driven assessment of airmanship within existing training environments, without disrupting established training protocols.
At the core of TACET-M is Cineon’s Empathic Learning Engine (ELE), a proprietary AI architecture that uses eye-tracking data to assess pilot behaviour and cognitive performance. This includes visual scanning, prioritisation, attention management and response to workload under pressure.
Using the Mixed Reality capabilities of the Varjo XR4 Focal Edition headset, session data is autonomously analysed and visualised, providing instructors with clear, actionable insight into how pilots respond to complexity, distraction and unexpected events.
The ‘Focal Edition’ provides high-resolution Mixed Reality (MR), gaze-driven autofocus passthrough, integrated eye-tracking and tracking capability suitable for cockpit MR training scenarios/environments.
Use of the XR4 Focal Edition headset ensured that:
- The headset and pass through cameras have a high resolution, refresh rate and suitable field of view
- High quality eye-tracking and other movement data can be captured
- The hardware allows for a low latency (for optimal user experience)
- The system is future proof
High-quality eye-tracking data enables the ELE system to detect subtle, real-time changes in eye behaviour. This allows the platform to identify different attentional modes and determine whether shifts in cognitive state are occurring during task performance. The ELE system has been trained on hundreds of richly labelled datasets collected across a range of task-relevant operational contexts. This enables its machine learning models to remain both accurate and task-specific, identifying meaningful behavioural signals rather than relying on indirect proxies.
Eye-tracking outputs, alongside additional behavioural and task performance data, are processed through bespoke performance frameworks tailored to each operational context. The system then generates a score/outcome/recommendation against key human factors, or airmanship qualities, relevant to the task being performed.
How eye-tracking enables this assessment:
- Situational Awareness
- Eye-tracking enables the assessment of Situational Awareness by measuring scan patterns, visual coverage and fixation behaviour between critical areas of interest.
- Mental Capacity
- Changes in eye behaviour can provide valuable insight into cognitive workload and attentional demand. Mental Capacity can be inferred from an individual’s ability to redirect attention and re-prioritise information as task demands evolve.
- Decisiveness
- The analysis of visual search and repeated checking behaviours provide insight into the information-gathering processes that underpin decision making.
- Communications
- Communication effectiveness is assessed primarily through analysis of communication content, timing, accuracy and procedural compliance rather than through eye-tracking alone.
- Resource Management
- Eye-tracking helps identify how effectively operators balance attention between internal cockpit tasks, external threats, mission objectives and supporting assets.
It is important to note that these assessments are not intended to replace the instructor. Rather, they help to reveal aspects of performance that would otherwise be difficult to observe directly. The system can therefore support instructor judgement by providing additional evidence, reducing the need to infer or guess what may have been occurring during critical moments. Notably, the outputs of the system will only be effective when the context to the task being completed is understood, where the instructor provides expert knowledge alongside sim data.
TACET-M in Action
Working with Ascent Flight Training as a delivery partner, TACET-M has been tested with trainees at RAF Cranwell during UKMFTS Phase 1 training. A cohort of students at Number 3 Flying Training School participated in a training exercise using the mixed reality headset within a Grob Prefect T1 FTD. The scenario was designed as a 15-minute simulation incorporating a series of progressively challenging events to evaluate the demonstration of airmanship qualities. A Qualified Flying Instructor (QFI) managed the fixed-base simulator, controlling the timing of fault initiations to align with the scenario’s requirements and providing expert view on the students’ actions.
The aim of this data collection exercise was to:
- To gather subjective feedback on the usability of MR as a training and assessment tool within a fixed-base simulator.
- To analyse eye-tracking data for identifying specific gaze patterns associated with competency, which will refine outputs in the advanced playback analysis software.
- To collect psychophysiological data to support machine learning models in assessing cognitive load, stress, and mental fatigue.
Specifically, we collected movement data related to the head, hands and eyes via the headset, as well as interactions with the cockpit.
What Are the Benefits of TACET-M?
The project was able to demonstrate the potential for substantial benefits both for defence training organisations, and for instructors and trainees. These were as follows:
- Objective Measurement of Airmanship: Eye-tracking and AI provide evidence-based insight into factors such as situational awareness, attention management, workload and stress, directly linking data to airmanship qualities that have traditionally been difficult to quantify.
- Earlier, Targeted Training Interventions: By identifying performance and cognitive challenges earlier in the training pipeline, instructors can intervene sooner, reducing the likelihood of costly late-stage attrition.
- Reduced Reliance on High-Cost Simulators: TACET-M enables more effective use of fixed-base simulators and XR technologies, easing pressure on full flight simulators while maintaining assessment quality.
- Consistent and Unbiased Assessment: Data-driven insights reduce subjectivity in evaluation, supporting more consistent decision making across instructors and cohorts.
- Scalable and Adaptable Architecture: The platform can be extended to additional aircraft, scenarios and safety-critical domains, including unmanned systems, and land or sea-based operations.