What is spatial computing, and how will mixed reality blur the lines between the physical and the digital realms?

This article has been updated.

“Take time to play video games with the kids in your life so you can start to understand what it truly means to be a generation that flows between the physical and the virtual.”

This is the advice Cathy Hackl, futurist and CEO of Spatial Dynamics, gives other business leaders to familiarize themselves with a future where the physical and the virtual worlds will converge. Or, to put it technically, they will need to understand the concept and implications of spatial computing or mixed reality, she says.

Hackl joined the World Economic Forum’s Meet The Leader podcast series to speak about the future of human interaction with technology. Here are some of the key themes that emerged from the conversation.

Spatial computing merges the digital and the physical world, overlaying our environment with virtual and augmented reality.

By using spatial computing, computers can naturally integrate into the physical world. Devices like Microsoft’s HoloLens and Apple’s Vision Pro headsets enable users to see the real world with digital objects superimposed that appear three-dimensional, blending with the real-world environment.

The headsets can turn virtual objects into holograms and make the experience more immersive through spatial sound and an understanding of the user’s surroundings.

Hackl describes spatial computing as the next evolution of computing, after the mainframe, desktop and personal computers and mobile devices.

“Mobile computing ushered in a huge sea change of opportunities and ways of interacting with technology,” she explained. “What you’re starting to see right now is an evolution from mobile computing into spatial computing – computing that extends beyond the screens. It starts to understand the physical world and expands computing further.“

Spatial computing encompasses various immersive technologies, including extended reality (XR), virtual reality (VR), augmented reality (AR) and mixed reality (MR). While these technologies share overlapping features, they differ in their implementation and user experience:

Spatial computing leverages these technologies along with advanced hardware like depth-sensing cameras and inertial measurement units (IMUs) to enable seamless interaction between digital and physical realms. This integration facilitates innovative applications such as immersive virtual field trips and interactive science simulations, revolutionizing fields from education to industrial design

Although spatial computing is still a young discipline, there are several examples of how mixed reality could be applied, both by consumers and in the workplace.

Gaming is one obvious application, with Marvel now allowing fans to become part of its superhero universe in an hour-long mixed-reality experience.

But spatial computing isn’t just for would-be superheroes. Nasa used it in construction of the Orion spacecraft, part of its Artemis program to take people back to the moon and, eventually, to Mars. When Lockheed Martin started to assemble the seats in Orion’s crew module, there was no sign of instruction booklets or drawings. Everything the technicians needed to know came from the HoloLens 2 devices they were wearing: voice commands talked them through every step of the assembly, with holographic images of the parts overlaid onto the seats to show where and how they needed to be mounted.

“They found that using spatial computing reduced labour costs and hours by 90%. That is an early example, a very unique case,” Hackl told the World Economic Forum. “It’s starting to prove that spatial computing can give us superhuman powers of sorts, augmenting our workforce.”

Similarly, there are now examples in the medical field, such as a fully functional, immersive model of a human heart. Using Apple’s Vision Pro headset, trainee clinicians can virtually interact with the life-like heart simulation that reacts just like the real thing.

Hackl also pointed to the Forum’s Global Collaboration Village, which encourages problem-solving through partnership, as a stepping stone towards mixed reality.

Four key components make up spatial computing. Hardware is one of them, not just mixed-reality headsets but also new technologies including wearable AI devices such as the AI Pin and Rabbit R1, says Hackl. She foresees an explosion of these over the next few months.

When it comes to the next component, software, she stresses the importance of computer vision to enable computers to understand objects and people around them

“It's really important to understand that everything that's been happening in the AI boom is what's starting to help spatial computing truly happen. Because you need these spatial computers to understand the physical world. They can only do that through computer vision. For that, you need to create and train large models to understand the world.”

Large language and vision models in AI, along with game engines, have been paving the way for this, Hackl adds.

Another key component in spatial computing is data, with new types of data emerging and others being replaced. She points to 2D pixels being replaced by 3D “voxels”.

“For all of this to work and actually happen, you're going to need connectivity at levels we have never seen. 5G is not even going to cut it. You're going to need 6G and everything in between.”

“Spatial computing isn't one single technology or one single device. Spatial computing is almost like a new technological field,” Hackl points out.

And, as AI has shown over the last few years, new technology paradigms come with new challenges.

“When you expand computing into the physical world, you also have to start thinking about something called virtual air rights.”

“Who owns the air around me? Who has the right to put an ad in front of me? Who has the right to put audio in it? That's where I think virtual air starts to become a bigger conversation that is going to impact absolutely everyone.”

But a big technological shift such as this also has implications for the type of talent companies need to exploit the opportunities opening up. Getting close to a generation that has grown up as digital natives will play a big part in succeeding at this.

“Generation Alpha is a generation I do a lot of work with and study very closely. My three children are Generation Alpha. To them, what happens in the virtual space isn’t less real than what happens in the physical space. They move between both realms.”

“When you look at Gen Alpha, they are more in the gaming space. They're very entrepreneurial. They are ‘world builders’. They have been building worlds in Minecraft and Roblox,” highlights Hackl. “That in itself puts a different spin on what companies need to do to hire professionals, retain them and train them.”

To attract the youngest customer demographic, business leaders will need to immerse themselves in the mixed reality world that comes naturally to Gen Alpha. Hackl recommends the proverbial “getting down with the kids”, joining them in their digital pursuits, to truly get to grips with what makes this generation tick.

The quotes in this article have been lightly edited for clarity.