Augmented and virtual reality: the promise and peril of immersive technologies

We are at the cusp of a major revolution from mobile to immersive computing. Last year was seen as the dawn of a third wave of devices employing augmented and virtual reality (AR and VR), which define the two spectrums of immersive technology that could replace mobile computing.

A range of major products came to market in 2016 from companies including Oculus VR, Sony and Google. Since it bought Oculus for $2.1 billion, Facebook has acquired a further 11 AR/VR companies, underscoring the company’s view that VR and AR will form the next frontier. The large investments and acquisitions by tech giants suggest that these technologies will become increasingly integrated with the platforms on which we consume content.

According to a recent estimate by Goldman Sachs, AR and VR are expected to grow into a $95 billion market by 2025. As the chart below shows, the strongest demand for the technologies currently comes from industries in the creative economy - specifically, gaming, live events, video entertainment and retail – but will find wider applications in industries as diverse as healthcare, education, the military and real estate over time.

Moving from observation to immersion

AR and VR will offer a completely new creative medium – “an artist’s dream to build worlds, pixel by pixel,” according to Drue Kataoka, an artist and technologist. This promises the replacement of rectilinear devices with technologies that depict worlds in ever-expanding concentric circles, providing a level of immersion and experience that has never been seen before. This could be game-changing: users will no longer view content but will be placed inside ever-expanding virtual worlds and find themselves at the centre, hence the “immersive” nature of the technology.

“We’ve effectively had the same flat screen medium since 1896. VR/AR uniquely provides a sense of presence and immersion, it’s a brand new art form and brand new form of experiencing,” says Eugene Chung, founder and CEO at Penrose Studios.

Reduced production costs in creative activities

“Virtual prototyping” allows us to shorten the time and cost of iteration in product development while also improving the quality of the end product. For example, one design firm passed $50,000 worth of savings to a client in the aeronautics industry by using VR prototyping to abolish two physical prototype cycles and eliminate the time that would have been required for the assembly of custom samples. Wider use of virtual prototyping will allow companies to reduce the number of costly prototypes needed, as well as significantly decreasing the timeline from conceptual design to production and commercialization.

“You can iterate on your city plan, your home and your construction worksite many more times before you actually start to dig or make a change. As a result, we’re going to get better creations,” says Jeffrey Powers, co-founder and CEO of Occipital.

While traditional technologies also allow companies to prototype, with immersive tech, designers are provided a more direct experience by being able to walk, fly and interact with their prototypes, either in a VR or AR environment. The implication is that immersive technologies promise higher accuracy in design and, as a consequence, an end product of higher quality at a potentially cheaper cost than traditional prototyping technologies can provide.

Lower barriers to entry for new creators

Immersive technologies will also empower smaller firms to produce higher quality content at lower cost. The technology already exists to process 360-degree imagery in hours – something that until recently would take days – and is within reach for filmmakers on small budgets. In the same way that smartphones and apps moved mobile photography outside the realm of professionals and enthusiastic hobbyists, we can expect AR and VR to open up new creative avenues for all of us.

As a tool for empathy and cognitive enhancement

Immersive technologies may also allow us to feel closer to global issues, such as humanitarian crises, enabling a form of telepresence that evokes levels of empathy as if one were present. According to Lynette Wallworth, an artist and director, AR and VR “provide a layer of authenticity of experience” not offered by other mediums.

Although there are suggestions that increased consumption of digital media can cause a decline in empathy, many artists working with AR and VR are convinced that the medium will become the “ultimate empathy machine”, fostering a society with informed perspectives of other communities and identities. Gabo Arora, founder and president of Lightshed and a creative director and senior advisor at the United Nations, explains: “You’re discovering a new grammar of storytelling and emotions”. If the optimists are right, we could be well on the way to a more informed and creative world.

There is also promise for VR and AR to provide immersive learning experiences. Beyond immediate gamified learning, VR and AR’s biosensors have been harnessed in therapeutic domains, allowing one man to drive for the first time in his life using just his brain. “[Immersive technologies] are a form of brain augmentation that networks our biological systems to a digital device,” says Tan Le, CEO of Emotiv.

Beyond technical challenges – ranging from device size to battery life – one potential barrier to rapid progress in the AR/VR industry is the lack of talent to meet the demand for growth. The industry is at “ground zero”, so it is difficult to show the gap in supply and demand of talent. However, initial informal measures exist, such as US data showing that demand for freelancers with VR expertise grew far faster than that for any other skill in the second quarter of 2017, a 30-fold, year-on-year increase. Similarly, a recent survey of 200 Canadian companies working on VR projects concluded that VR will face a talent crunch that “could fuel consolidation between companies”.

Strategically developing domestic talent

Governments would be wise to develop strategic planning that captures talent in emerging technologies to guarantee their countries are at the leading edge of the next computing frontier. China is one example of an early mover and, according to Di Yi, vice-president of Perfect World Co., “the Chinese government offers substantial support for the VR industry.”

Areas such as Zhongguancun, Beijing, are subsidizing companies by up to $1.45m to further develop the VR industry and position the region as the next global technological hub. Other locales, including Beidouwan VR Village, in Guizhou province, offer grants to support content development and investment. By 2019, the village is expected to produce 1.5 million pieces of VR-related hardware, as well as 500,000 transactions of software content – delivering 3,500 new jobs in the process.

Enticing foreign talent

Governments can also take an active role in investing in immersive content. A 2016 survey of 500 AR/VR professionals reported that almost 50% are using their own personal funds to develop their companies. Fewer than 8% reported ‘other’ sources of funds, including government. Given that VR production costs can run into the hundreds of thousands, forward-looking policy-makers are promoting subsidies to entice VR talent from around the world. The canniest ensure that, in the process of doing so, they protect and develop their own industries at the same time.

In France, for example, the government-backed CNC Fund provides funding for VR/AR producers to co-produce content with local teams, offering grants that cover both development and production. In one recent example, the fund supported around 40% of the €500,000 costs required to produce a VR short film. The CNC also has the authority to award a tax rebate of up to 30% of qualifying expenditures to projects wholly or partly made in France and initiated by non-French companies.

Do these approaches work? While the effect on VR content is still being evaluated, the policy appears to have been effective in supporting ‘traditional’ filmmaking in France. Around a year after implementation, 31 projects from eight different countries had been launched, compared to only four in the previous year. These projects, which included blockbusters such as Inception, were estimated to generate approximately €119 million in direct spending in France and involve 450 days of filming – compared to €7.4 million in spending and 84 days of filming in the year leading up to the policy. Considering the nascent stage of VR, it is too early to tell whether the gains can be transferred to this new industry, but the attractiveness of the mechanism to creators is clear.

Designers of software have an incentive to keep users inside of their websites and apps because their business models increasingly rely on the collection of personal data as a way to personalize content.

Adam Alter, associate professor of marketing at NYU’s Stern School of Business, describes the strategy as a “brute force, big data-driven” approach: “Companies A/B test different features of a product and iterate to the point that is maximally difficult to avoid”. In addition, deep-pocketed companies have “teams of psychologists working according to the latest information to make humans more engaged”.

Indeed, evidence from the last decade shows that, while our overall leisure time is increasing, we are spending more of it using screen-based devices. A key driver of this shift is the practice of taking engagement as the main success metric for digital technology. The more time we spend on a device, the more data there is to collect about our interactions and the more targeted product offerings become – a reinforcing cycle that is likely to speed up as immersive technologies enter the mainstream.

The result is that the content we experience in immersive technology will be increasingly personalized. This could play out in a number of ways, but advertising, in particular, is well-placed to benefit.

Advertising promises to be more targeted

Advertising is already increasingly personalized as our personal data allows better and better targeting. In the context of immersive technologies, the term “gaze-through rate” has been coined to describe the effectiveness of an augmented or virtual advertisement in capturing user attention. Companies such as Retinad offer analytics to track behaviour on VR/AR devices with an aim to increase the conversion rate from advertisements as well as user engagement with content. Thus far, they have been shown to be up to 30 times more effective in engaging users than mobile advertisements.

Greater engagement may negatively impact well-being and privacy

The drive to capture our attention creates two challenges. First, our well-being is at stake: non-screen activities are more clearly linked to happiness than not. One longitudinal study of a major social network found a negative association with increased engagement and individual well-being, suggesting “a possible trade-off between offline and online relationships”. Teenagers in the US who devote just six to nine hours a week to social media are 47% more likely to say they are unhappy than those who use social media even less.

Secondly, a lack of sovereignty over personal data may push users away from the long-term adoption of new technologies. A report by the World Economic Forum shows that 47% of people across six countries have stopped or avoided using a service because of inadequate user controls and this figure rises as high as 70% for China. This suggests that user privacy and data controls are a key concern for consumers. Given the enhanced data tracking features from immersive technologies, from tracking eye-movements and facial expressions to haptic data (relating to the sense of touch), the personal data at risk will become more intimate than ever making worries about user privacy a far more serious concern.

Regulatory frameworks

The privacy concerns relating to traditional media are already surfacing in immersive content. If developers are unwilling to provide clear and agreeable terms of use, regulators must step in to protect individuals – as some jurisdictions have already.

The best place from which to draw inspiration for how immersive technologies may be regulated is the regulatory frameworks being put into effect for traditional digital technology today. In the European Union, the General Data Protection Regulation (GDPR) will come into force in 2018. Not only does the law necessitate unambiguous consent for data collection, it also compels companies to erase individual data on request, with the threat of a fine of up to 4% of their global annual turnover for breaches. Furthermore, enshrined in the bill is the notion of ‘data portability’, which allows consumers to take their data across platforms – an incentive for an innovative start-up to compete with the biggest players. We may see similar regulatory norms for immersive technologies develop as well.

Providing users with sovereignty of personal data

Analysis shows that the major VR companies already use cookies to store data, while also collecting information on location, browser and device type and IP address. Furthermore, communication with other users in VR environments is being stored and aggregated data is shared with third parties and used to customize products for marketing purposes.

Concern over these methods of personal data collection has led to the introduction of temporary solutions that provide a buffer between individuals and companies. For example, the Electronic Frontier Foundation’s ‘Privacy Badger’ is a browser extension that automatically blocks hidden third-party trackers and allows users to customize and control the amount of data they share with online content providers. A similar solution that returns control of personal data should be developed for immersive technologies. At present, only blunt instruments are available to individuals uncomfortable with data collection but keen to explore AR/VR: using ‘offline modes’ or using separate profiles for new devices.

Managing consumption

Short-term measures also exist to address overuse in the form of stopping mechanisms. Pop-up usage warnings once healthy limits are approached or exceeded are reportedly supported by 71% of young people in the UK. Services like unGlue allow parents to place filters on content types that their children are exposed to, as well as time limits on usage across apps.

All of these could be transferred to immersive technologies, and are complementary fixes to actual regulation, such as South Korea’s Shutdown Law. This prevents children under the age of 16 from playing computer games between midnight and 6am. The policy is enforceable because it ties personal details – including date of birth – to a citizen’s resident registration number, which is required to create accounts for online services. These solutions are not infallible: one could easily imagine an enterprising child might ‘borrow’ an adult’s device after-hours to find a workaround to the restrictions. Further study is certainly needed, but we believe that long-term solutions may lie in better design.

Rethinking success metrics for digital technology

As businesses develop applications using immersive technologies, they should transition from using metrics that measure just the amount of user engagement to metrics that also take into account user satisfaction, fulfilment and enhancement of well-being. Alternative metrics could include a net promoter score for software, which would indicate how strongly users – or perhaps even regulators – recommend the service to their friends based on their level of fulfilment or satisfaction with a service.

The real challenge, however, is to find measures that align with business policy and user objectives. As Tristan Harris, Founder of Time Well Spent argues: “We have to come face-to-face with the current misalignment so we can start to generate solutions.” There are instances where improvements to user experience go hand-in-hand with business opportunities. Subscription-based services are one such example: YouTube Red will eliminate advertisements for paying users, as does Spotify Premium. These are examples where users can pay to enjoy advertising-free experiences and which do not come at the cost to the content developers since they will receive revenue in the form of paid subscriptions.

More work remains if immersive technologies are to enable happier, more fulfilling interactions with content and media. This will largely depend on designing technology that puts the user at the centre of its value proposition.

This is part of a series of articles related to the disruptive effects of several technologies (virtual/augmented reality, artificial intelligence and blockchain) on the creative economy.

In addition to the experts quoted, the authors would like to thank the following constituents for their input and comments on this article: Aaron VanDevender, Chief Scientist and Principal, Founders Fund; Allen Yang, Professor, University of California, Berkeley; Deepak Krishnamurthy, Executive Vice-President, Chief Strategy Officer, SAP; Geng Danhao, Senior Vice-President, iQIYI.com; Jonathan Dunn, Partner, McKinsey & Company; Masa Inakage, Professor, Keio University; Matt Weiss, Managing Director, IDEO CoLAB; Masaru Sugiyama, Vice-President, Goldman Sachs.