How VR is empowering disabled musicians

Most new digital technologies tend to be designed with an able-bodied user in mind. The first desktop computers required fine motor skills to navigate software menus using a mouse, and mobile phones need users to press buttons, swipe screens, and so on. To use such technology a person needs to be fairly dexterous.

In our Performance without Barriers research group, we design digital musical interfaces with disabled musicians in mind. This work engages disabled performers from the start of a new technology and looks at the specific abilities they have. In this way, technologies develop in tandem with them, taking into account their mobility, needs and creative interests.

Current VR technology is designed for the able bodied, but more importantly it often allows only for passive interactions – listening to music performances, such as Elton John’s 360 concert, for example, or “riding” a rollercoaster. We were more interested in how disabled musicians can use VR technologies in an active and performative way.

Working together

Our research team consists of a diverse group, including electronic engineers, computer scientists, sonic arts researchers, immersive content designers, a soloist ensemble and a local group that helps disabled musicians perform and compose their own music independently. Together with these musicians, we teamed up with a US software developer, who was designing a VR musical instrument called the “Infinite Instrument”, running on a 360° VR headset called HTC VIVE.

The instrument was developed with able-bodied musicians in mind, so we designed it to take into account different types of mobilities. This led to one of our musicians with cerebral palsy playing a new VR instrument that was specifically designed to take into account her expressive upper body movements. It did not require her to use fine motor precision in her arms or fingers, which she does not possess.

VR headsets are necessarily about what you see. However, we found that the tactile feedback from the hand-held controllers – through which a user accesses a menu and press commands to reach content – allowed this particular musician to play the instrument by feeling and hearing it, rather than seeing it through the headset. Not having to rely on the headset also meant she could maintain visual contact with other musicians during the performance.

Another example of our collaborative VR design was working with a blind performer. Key to this musician were his virtuosic skills on the clarinet. The idea was to build on his expertise and adapt and enhance his musical performance skills using VR technologies.

VR experiences are so concentrated on the visual experience that often they disregard the inherently immersive nature of sound, but it’s all around us, a proper 360° experience. Clearly, a visual headset was not relevant or helpful to this blind musician. We needed to focus on how we could use the other immersive qualities of a space for a music performance.

Our team is based at the Sonic LAB in Queen’s, an immersive and fully customisable 3D sound space – often referred to as the iMAX for the ears – we decided to adapt the VIVE technology to this existing context.

For our clarinettist we focused on the VIVE controllers, as they are more tactile than visual. They can also track physical positions in a space, which meant the clarinettist was able to position sounds in the 3D Sonic LAB by having the VIVE controller strapped to his instrument.

Inclusive thinking and design

These bespoke VR instrument designs featured in a showcase concert in November 2018, where disabled musicians performed alongside musicians from the Hard Rain Soloist Ensemble.

The audience was positioned in a circle closely surrounding the musicians in order to enhance the immersive experience of the performance space.

The disability equipment market worldwide is estimated to increase to more than US$8 billion by 2020, and sales of disabled equipment in the UK have increased over 93% over the last ten years. All technologies, including VR, can be inclusive if the perspective of disabled people is part of their design.

Our approach reveals how new technologies can be developed, that actively engage disabled musicians in music making and demonstrate a commitment to quality of life for disabled musicians. We will continue to design instruments that can be used in VR, but we will now focus on using more affordable systems, with a view to creating a virtual reality ensemble of disabled and able-bodied musicians.