How virtual reality could help save your life

In a landmark operation at the University of Minnesota in 2017, surgeons successfully separated conjoined twin sisters using an unprecedented tool: virtual reality.

To prepare for the complex and risky procedure of separating the sisters, researchers combined high-resolution images from MRIs and CT scans to create a three-dimensional digital model of the infants’ intertwined hearts. Donning virtual reality goggles, doctors navigated the walnut-sized organs, identifying critical anatomical defects and weighing approaches for performing the complex separation. Based on what they discovered, the surgeons altered their original strategy, adopting an alternative approach for the 9-hour procedure that is credited with saving both young lives.

It may sound like science fiction. In fact, “digital twins” – computational models of patients’ hearts – are becoming part of standard medical care. They can be viewed on a computer, can simulate and analyze cardiovascular physiology, and can be created as objects using 3D printers. Cardiologists now routinely use 3D imaging to visualize blockages in arteries before they operate to clear them. To correct serious congenital heart defects in infants, cardiologists at Great Ormond Street Hospital for Children in London have begun to use digital recreations of children’s tiny hearts to plan the operation in advance.

Digital Personal Avatars

Since the best way to understand how a patient’s heart is working is to see it in action, prototypes of a beating “virtual” heart are in development. Being able to visualize the body has long been at the heart of medical discovery. Today, advanced technologies offer bold new ways to see and to discover. Inspired by the astonishing computer-generated images used in films, a technique called “cinematic rendering” combines raw data from CT and MRI scans and then uses algorithms to simulate the complex play of light and shadow in three dimensions, even modeling how light penetrates living tissue. The result: 3D renderings of bone, muscle, organs, blood vessels and nerves that physicians can navigate in and around, seeing in a way that has never before been possible.

The heart was the first organ to be precisely modeled this way, but digital twins of other organs, including the brain, are being developed. Eventually, we’ll have complete whole-body digital twins of individual patients – in effect, digital personal avatars.

The applications are wide-ranging. Cancer surgeons will be able to evaluate precisely how tumors are positioned in relation to healthy tissues. Orthopedic surgeons will be able to use advanced 3D images to visualize the topography of complex fractures. In a pilot study at the Cardiff University Brain Research Imaging Centre, scientists are using cinematic rendering to study nerve fibers in the brain linked to multiple sclerosis.

Dedicated computational models allow us to create digital twins at both the macro and the micro level, visualizing the shape and structure not just of organs, but also simulating bio-molecular processes at a cellular level, including proteins that make up cells, and even genes. The latest imaging can even detect metabolic changes in cells, making it possible to distinguish cancer cells from healthy cells.

Predictive modeling

Digital personal avatars, in other words, will be far more than fancy anatomical models. They will incorporate a patient’s cellular, molecular, genetic and clinical information. The state-of-the-art of personalized medicine, they will provide physicians with deep insights into the specific features of an individual patient’s condition. By knowing a patient’s genetic and molecular make-up in advance, doctors can determine whether a particular medication is likely to help and what dosage to use. Predictive modeling will make it possible to detect health problems long before they show up clinically. Data gathered across a patient’s lifetime will provide profound insights into aging and health.

All of this is possible because of our ability to analyze vast amounts of digital information using artificial intelligence and machine learning. To give a sense of the scale of the data we have at our fingertips: Siemens Healthineers has compiled a database of more than 100 million images, collected from MRIs, CTs and other imaging technologies and annotated with relevant clinical information, which are being used to train AI algorithms. The aim is to develop algorithms that will help doctors better diagnose and treat a wide range of diseases.

Share - but protect - data

The more data we can mine, the more we will learn. But managing sensitive health data is a complex task. At the moment, there are significant competitive, regulatory, and logistical barriers to sharing medical data. One approach is the creation of digital “ecosystems” open to partners who agree to share data, using common storage and communications systems.

As individual patients and as a society, we will reap enormous value by sharing health data. But in the era of digital personal avatars, how can we protect an individual patient’s confidentiality and still make data available for the common benefit of better healthcare? The success of the cryptocurrency Bitcoin offers a potentially useful model. Bitcoin uses blockchain technology, which protects data by dividing it into discrete blocks and storing them in distributed databases that are not connected to a common processor. Only those with an encrypted key -- the patient and the doctor, for example – can access the data. Others can share discrete parts of the data only when they have permission. Blockchain technology would make it possible to associate digital twins with their respective human counterparts, while at the same time sharing the data anonymously with third parties.

Digitization helps to save lives

A decade ago, the idea of surgeons donning virtual reality glasses to navigate through a digital rendering of a patient’s heart would have seemed far-fetched. A decade from now, digital twins will be a standard tool in medicine, and wonders unimaginable today will be on the horizon. The Nobel prize-winning nuclear physicist Niels Bohr famously quipped, “It is very hard to predict, especially the future.” One thing we can say with certainty: Digitizing healthcare will expand precision medicine, transform care delivery, and improve patient experience. And it will help healthcare providers achieve better outcomes at lower costs. Our ability to model an individual patient’s body at virtually every level, from the structure of the heart to the shape of a protein on a cancer cell, will lead to profound new insights into biomedicine – and save lives.

Two little girls named Paisleigh and Paislyn Martinez, born as conjoined twins and successfully separated thanks to advanced 3D imaging, are living proof.