This 'smartphone for surgeons' could be the future of operations

One day in mid-December, orthopedic surgeon Dr. Bruno Gobbato walked into an operating room in Jaraguá do Sul, Brazil, put on a HoloLens 2 mixed-reality headset and prepared for surgery.

Joining him remotely were fellow surgeons Professor Thomas Gregory, who was tuning in from Paris, and Dr. John Erickson, who is based in New Jersey. Gobbato’s patient had a collarbone fracture that hadn’t healed properly, so Gobbato needed to reposition the bone and perform a shoulder arthroscopy, which involved inserting a small camera into the joint to try to determine what was causing the man’s shoulder pain.

Gregory and Erickson were linked to Gobbato’s headset via the Microsoft Dynamics 365 Remote Assist app and shared his field of view on their computer screens through Microsoft Teams. They could see the patient and the holographic images Gobbato generated from a CT scan, one showing the patient’s damaged clavicle and another replicating his healthy clavicle. The three surgeons on three continents discussed how to approach the procedure, conferring on each step and sharing their respective approaches.

“They were my partners helping me with the surgery,” Gobbato says. “We had a French perspective, we had an American perspective and we had a Latin American perspective. We had one-quarter of the world inside the operating room.”

The procedure was part of a project aimed at showing how HoloLens 2 can benefit surgeons and enable collaboration between doctors worldwide. Conceived by Gregory, who in 2017 performed the first surgery using the original HoloLens, the initiative involved orthopedic surgeons from 13 countries on five continents.

Between November and January, 12 of the doctors used HoloLens to perform a surgery — ranging from a knee procedure in the United Arab Emirates to a shoulder replacement in South Africa — while the others, joined by Gregory, took turns sitting in to observe and offer input. The project culminates today with a series of roundtable discussions and interviews around the intersection of mixed-reality technology and its impact on health care.

For Gregory, the project was a way to convey the benefits of what he sees as “the smartphone for surgeons.” Many people rely on smartphones to quickly access information in their daily work, he points out, but surgeons can’t use phones or computers in a sterile operating room setting. They typically plan procedures in advance and have little access to patient data or other resources once a surgery begins.

When the first HoloLens was introduced in 2016, Gregory immediately saw the potential for surgeons to use it as a tool for information and communication. Suddenly, he could use hand gestures or voice commands to call up information instead of touching a keyboard or mouse.

“It was what I was looking for for many years,” says Gregory, chief of the department of orthopedic and trauma surgery at Avicenne Hospital in Paris and a professor of orthopedic surgery at the University Sorbonne-Paris-Nord.

“I realized that I was able to use the HoloLens as a computer or a smartphone to get any information I need when I need it, during surgery. That allows surgeons to be quicker, to be more efficient and to improve performance.”

Gobbato, a self-described “tech guy” and “geek,” was so anxious to get a HoloLens that soon after the device was released, he hopped on a plane and flew eight hours to Miami to buy one. He has been using the device in more complex surgeries for several years.

“I really think mixed reality will be used in the operating room for so many things in the future,” Gobbato says. “I’m very excited to see what we will be able to do with it in the next few years.”

HoloLens is operated with hand gestures and voice commands, enabling surgeons to view three-dimensional holographic images of a patient’s anatomy created from X-rays or other scans. Surgeons can move those virtual images around to see them from different angles. They can also use the HoloLens to access patient data during surgery, call up videos or documents to help solve problems and contact other specialists for advice.

Dr. John Sledge, an orthopedic surgeon in Lafayette, Louisiana, performed one of the procedures that was part of the HoloLens project, a 10-hour spinal fusion surgery. As Gregory and two surgeons from the U.K. and the UAE looked on, Sledge pulled up X-rays and scans of the patient’s lower back to locate the pieces of hardware that needed to be removed and determine how to best position his instruments to access them. He had also loaded images taken shortly after the patient’s accident so he could explain the preop history to the other surgeons.

Without the HoloLens, Sledge says, he would have been limited to a few images from the patient’s scan on a computer screen.

“It’s an enormously limited data set that’s available to me,” he says. “With the HoloLens, I can pull up the images I want and make them bigger as needed. I’ve got all the images I need literally right in front of me, in whatever size and clarity that I need them to be.”

HoloLens is also useful for planning surgeries and training, Sledge says. If he’s planning a shoulder operation, for example, he can create a holographic representation of the patient’s shoulder to determine where an implant should be placed or whether the bone needs to be reconstructed.

Instead of printing out a 3D replica of a bone and having just one shot at practicing an operation on it, Sledge says, he can test his approach on a hologram as many times as needed. For training, he can generate three-dimensional interactive models of any surgical procedure or even create simulated complications on a hologram — say, a fracture or unexpected bleeding — for doctors in training to solve.

“Medicine, particularly surgery, is still an apprenticeship. You watch a person operate 100 times before you’re allowed to,” Sledge says. “But now we can have residents run through 100 operations on the HoloLens, complete with rare complications and their solutions. We can do worst-case scenario training. With the HoloLens, we can make a problem occur and the doctor in training has to solve it.

“We can standardize the surgical training so that all graduates across the world will have seen and learned how to solve all of the cases in the HoloLens library in their field of training,” he says. “There’s a huge difference between you watching someone else solve a problem and you having to solve it yourself.”

Orthopedic surgeon Dr. Stefan Greiner had never used the HoloLens before agreeing to participate in the project with Gregory, a good friend of his. Greiner was initially skeptical about the device, seeing it as “another gadget” that might be useful in rare cases.

But as he learned how to use it, Greiner started to see more possibilities. He envisions using the device to display images during an arthroscopic surgery to avoid having to look back and forth at a screen, where images from the scope are usually displayed, while he’s operating.

“I think it’s going to improve the precision of surgery and the safety of surgery, because you don’t have to look somewhere else,” says Greiner, who runs the shoulder and elbow department at Sporthopaedicum Regensburg, one of the biggest sports medicine facilities in Germany.

The device could also be used to help prepare patients for surgery, he says, by explaining procedures through holograms or animations.

“It’s so much more clear if you can see holograms in action, rather than just see a small picture,” he says. “This would be of great use for patients to understand what kind of surgery they are going to have.”

Dr. Ashish Babhulkar had also never used a HoloLens before signing up for Gregory’s project. Babhulkar, head of the shoulder and sports injuries department at Deenanath Mangeshkar Hospital & Research Center in Pune, India, did two reverse shoulder replacement surgeries — in which the ball and socket position of the shoulder joint is reversed — in collaboration with Gregory and an orthopedic surgeon from the U.K.

With input from the two doctors, Babhulkar superimposed a holographic image of the socket implant on top of the patient before inserting it to get a perfect fit. His field of view also contained a surgical manual with 3D videos for each step, which the surgeons referenced to see a device that Gregory used for shoulder surgery.

Though Babhulkar didn’t need the manual for the procedure, he saw how useful HoloLens could be as a training tool.

“Imagine that tomorrow in Kenya, someone’s got a HoloLens and he wants to start doing this surgery. He doesn’t have to come to India to learn it,” Babhulkar says. “We can walk him through the process without him making errors and help build his confidence. It’s going to revolutionize treatment in some remote areas.”

The experience, he says, convinced him that HoloLens will be a critical tool in medicine going forward.

“The device is so powerful that as we progress on it, every year we are going to innovate and add novel applications to it,” he says. “Hopefully, we will be using it for all our cases eventually.”

The HoloLens and its precursor, the Kinect line of motion-sensing devices, were created by Microsoft Technical Fellow Alex Kipman, a visionary engineer who saw a need for a device that merged the physical and digital worlds in a meaningful, three-dimensional way.

Kipman sought to develop a device that could understand humans and their environments and transform the digital world from one of flat rectangles of glass — computer screens, phones — to a universe of holograms that can interact with or simulate the real world.

The first HoloLens was released in 2016, followed by HoloLens 2 in November 2019. The device is equipped with sophisticated environmental mapping hardware and an eye-tracking camera that enable it to understand the space it’s being used in and what the user is focused on. The Microsoft Dynamics 365 Remote Assist application was designed to allow HoloLens to connect with Teams so users can collaborate with people worldwide.

A decade ago, it might have been hard to imagine surgeons using holographic images for surgery or envision what mixed-reality technology could enable in the medical field. The HoloLens 2 is now being used for a range of applications in health care, from virtual hospital rounds for medical students to treating patients with COVID-19, and a “HoloPatient” used for simulations of various medical conditions to train doctors and standardize treatment.

The use of HoloLens in surgery is new, propelled by Gregory and other pioneering physicians, says Elena Bonfiglioli, Microsoft’s managing director and regional lead for health care and life sciences in Europe, the Middle East and Africa.

“These leaders are reimagining surgery, which has been done in a traditional way for a very long time,” she says. “The ability to imagine change and implement change is what is so transformational in these clinicians. We are really at the beginning with using HoloLens for surgery, but the path is charted and the change agents are there.”

The power of HoloLens, Bonfiglioli says, is its ability to superimpose objects and data on the real world in a trusted digital environment and enable a natural means of learning.

“The future of medicine is empowered by holographic computing, because holographic computing gives the new dimension of visual,” she says. “We have had the dimension of collaboration for a long time. We’ve had the dimension of data, powered by artificial intelligence. But we didn’t have something that is very typical to the human, which is visual enhancement. That’s why it’s so powerful. It’s a technology that in itself is a change agent, empowering those who can see the future.”

Gregory’s department has been using HoloLens 2 for all surgeries for more than a year and the device has become part of his daily practice. Younger colleagues will call him on HoloLens for advice if they encounter issues during a surgery. Gregory might call another specialist for input during a surgery or contact a nurse to ask for the next patient to be brought in.

In January 2020, Microsoft partnered with Gregory and others to launch Institut Moveo, a research and teaching center dedicated to artificial intelligence in surgery. The center, located at University Sorbonne-Paris-Nord, brings together computer scientists, mathematicians and surgeons and provides training on using HoloLens for surgery.

Future plans include collecting HoloLens data from surgeries and using it to develop algorithms that could improve surgical procedures or even identify, based on eye movements, when a surgeon is encountering difficulty.

Gregory sees HoloLens as a device that, at a much lower cost than the surgical robots used for some types of minimally invasive surgery, which can run about $2 million, can make artificial intelligence in surgical practice accessible to even small medical facilities around the world.

“With HoloLens, we’ve got a tool to democratize this digital revolution in all of the operating theaters around the world,” he says. “The impact of this is massive. I think in the future, surgeons will not be able to perform a surgery without HoloLens.”