Using hybrid reality to deliver Human Factors insights sooner

Human Factors (HF) is an integral part of successfully delivering any new rail vehicle to market. Compliance with ergonomic, usability and accessibility legislation and best practice needs to be addressed and evidenced in a fully integrated fashion throughout the entire development programme. A range of tools and techniques is available to achieve this, many involving the observation and assessment of trial users interacting with physical rigs and mock-ups. These typically increase in fidelity as the train development progresses, most commonly culminating in a full size representative mock-up that closely emulates the production vehicle. Design changes at this late stage in a project can be costly and time consuming, so resolving as many potential HF issues as possible before the final mock-up build is essential.

To address this, we have recently been exploring the use of new technology to create hybrid prototypes, which combine basic, low-fidelity physical mock-ups with virtual reality (VR) and pose estimation to deliver a higher fidelity experience of the train and meaningful HF outcomes much earlier in the development programme. These hybrid prototypes are fully immersive, highly flexible and offer the potential for larger numbers of subjects to experience and interact with a range of proposed design solutions.

The video gaming industry has transformed what can be achieved using relatively standard high-end desktop computers and low cost VR headsets. This creates a fantastic opportunity to combine the benefits of basic prototyping and VR to create an environment in which we can interact with a concept proposal from the very start of the design journey.

In a recent example, DCA used hybrid prototyping to analyse specified train driver tasks and assess comfort and reach parameters for a new cab design and desk layout. However, for the purposes of this article we have used images of a cab that is already in the public domain to illustrate our process. We were looking to see whether the driver could comfortably reach the controls used most commonly during driving operations without repetitive stretching and overreach, which can cause strain to the lower back and generate work-related stress injuries.

Traditionally, train cab design, and the driver’s desk layout in particular, starts with the use of basic 2D reach envelopes to map out control layouts, before progressing to physical mock-ups. By definition, this initial 2D approach provides a limited view of the physical design. Now, with hybrid prototyping, we can perform this early reach analysis in a representative 3D VR environment, which gives a much more realistic and thorough assessment of whether the design is suitable and fit for purpose. This potentially removes the need for some of the physical mock-ups and allows us to rapidly trial multiple design variants. Some of these may be bolder than normal because the VR environment allows us to assess innovative solutions with far greater confidence. We can also include larger numbers of users in our hybrid reality HF trials, and even scale the VR environment to allow any user to experience interactions as they would feel for a different sized user.

For this specific hybrid HF assessment, we synchronised the virtual train driver’s cabin with a simplified physical model of the driver’s desk and seat. An HTC Vive Pro VR headset was used with Valve Index controllers for the hands, which we selected specifically for their individual finger tracking capability. This allows the user to interact with buttons and controls in a more realistic, natural way.

The actual driver’s chair swivels between defined detents, so we wanted to ensure that this was reflected in the virtual world as well. To achieve this, we added a Vive Tracker 2.0 to the back of the chair and paired this with a virtual chair in the VR world. We have found from experience that adding tracked elements, which the user can interact with, increases the sense of immersion significantly.

Another key feature of our hybrid model for this project was the use of Google’s Media Pipe machine learning software as the platform for our pose estimation model. This looked at live image feeds from a webcam and worked out where the ‘bones’ and ‘joints’ of the test subject were. You can see in the image below how the ‘bones’ and ‘joints’ are overlaid in real time onto the captured webcam image.

The final hurdle to overcome was integrating the pose estimation data with the VR model within Unreal Engine, the software we use to produce and run our VR applications. We have found that web sockets, with a Raspberry Pi for the MQTT messaging service, work well for this task. The system is set up in a truly scalable way such that if we want to add more PCs to perform more pose estimation calculations we can. It also allows us to exchange and combine data from both Unreal Engine and Unity (another game engine) and even from Arduinos (small consumer electronics boards). Essentially, we can simulate any kind of interaction, whether it be physical or digital, in VR.

With the generated data inside Unreal Engine, we then needed to make it meaningful for our HF assessment. We used the movement of the subject’s shoulders from an initial ‘comfortable’ seated position to infer how far they were leaning or twisting in the seat. We added simple interactions to all the train controls and buttons such that when the user touched them in the virtual world they illuminated. The colour in which they illuminated was directly linked to how far the user was leaning or twisting at the time. If the user isn’t leaning during driving operations, all the buttons would illuminate green, but if the user were leaning or stretching significantly then the buttons associated with that task would go amber.

After a user trial was completed, we captured a final snapshot from above of the whole driver’s desk to provide a clear colour map that summarises which controls require a posture change to reach them. The results from trials for three users of very different statures and reaches are shown above.

The image for the very large subject (99th percentile male; 6ft 4in or 193cm tall) shows a mostly green control array, indicating that they were able to reach almost all the controls without adjusting their posture. However, even with this extended reach there are still some yellow controls on the left-hand side of the desk, indicating that a small posture change was required to reach these low frequency HVAC and external lights controls. The average sized user (42nd percentile male, 5ft 9in or 175cm) user shows many more yellow controls on both the left and the right, highlighting the need to adjust the posture to reach the less frequently used controls, while the smallest user (11th percentile female; 5ft 1in or 155cm) user shows that more controls required a posture adjustment.

This data provided us with the necessary insight to improve or verify the design. Each control that requires a posture adjustment should be carefully considered, based on its frequency of use and criticality. Importantly, the hybrid prototyping technique allows us to make these changes and re-assess the outcomes early in the design process more quickly and at a lower cost than would have been possible with a full mock-up. Furthermore, it provides an excellent objective record of what is often perceived to be a rather subjective assessment.

This is just one example of how DCA is using hybrid prototypes to both accelerate our design processes, improve design outcomes and de-risk future development programmes. We are finding the tools that games engines now provide, when run alongside artificial intelligence and machine learning, allow us to simulate a whole host of scenarios and address questions that were previously very difficult to answer until very late in the development process.