0370 270 6000

already registered?

Please sign in with your existing account details.

need to register?

Register to access exclusive content, sign up to receive our updates and personalise your experience on brownejacobson.com.

Privacy statement - Terms and conditions

Forgotten your password?

Augmenting reality - a general overview

20 October 2016

This article was originally published on Lexis®PSL and the interview conducted by Alex Heshmaty. For more information on Lexis®PSL, please visit http://www.lexisnexis.co.uk/en-uk/products/lexis-psl.page.

IP and IT analysis

As part of our augmented reality (AR) series, Joe O’Callaghan, associate at Browne Jacobson, gives a general overview of the developing field and some of the emerging legal questions.

What is AR and how does it differ from virtual reality (VR) and mixed reality (MR)?

There is some debate as to how the concept of AR should be correctly defined, however in broad terms it refers to technology that allows us to experience computer generated information superimposed upon the physical world around us in real time, such that we experience a composite ‘augmented’ reality.

Core examples of AR platforms include headsets and other mobile or wearable devices, which superimpose visual data in the user’s field of view. However, this is not the only way that a user’s experience may be augmented by AR and other digitally generated information such as sound or simulated touch (haptic) stimuli, may equally be triggered.

To illustrate, the Pokemon Go app, which recently brought AR into the public consciousness, incorporates a very limited form of AR (Pokemon creatures shown superimposed on a live video feed of the user’s surroundings). In terms of consumer AR platforms, readers may recall the abortive ‘Google Glass’ smart glasses product, which Google ceased to produce in 2015, although it has been rumoured that this device may be resurrected in the future. Future consumer AR device launches will include the Microsoft Hololens 3D holographic headset, which is currently shipping in developer kits, as well as the headsets under development by Meta and the Google-backed company, Magic Leap.

Both Microsoft and Magic Leap prefer to refer to ‘mixed reality’ to describe the experience their products offer. MR is essentially a form of AR where virtual objects credibly inhabit the physical space around the user, offering a more immersive experience than may be achieved by other forms of AR, where physical and virtual objects are harder to distinguish.

In contrast with AR and MR, the more familiar concept of VR refers to an entirely computer generated virtual environment which a user ‘inhabits’ using a VR device. A key related concept here is ‘presence’, where a user experiences such a degree of immersion in the virtual environment that, to a greater or lesser degree, they experience the illusion of actually being physically present in the simulated environment.

What are its current/most common applications? Have any of these proved controversial?

Applications

There are all manner of different applications of AR in the pipeline. There is growing interest in this technology category in education, healthcare, logistics, engineering, military, and marketing contexts to name but a few.

In terms of current applications, visual search is one field that is already seeing significant investment and is already relatively well established. The technology of Blippar, for example, which is a UK business that has received substantial funding over recent years (most recently $54m in series D venture capital funding), allows real time visual search, enabling users to unlock digital content about products and other objects using a phone or tablet’s digital camera. The technology has already been used by a wide variety of brands for marketing purposes over a number of years, but the company has recently pivoted to focus on more general ‘visual search’ allowing users to search for details on all manner of different things, including, for example, album covers.

The retail sector has been an enthusiastic adopter of AR applications. The digital creative studio Holition has been the creator of some high profile examples. In 2012 it partnered with Uniqlo to produce the software for ‘magic mirrors’ at Uniqlo’s new San Francisco store, allowing consumers to virtually try on the full range of colours for a range of fall and winter jackets. Its white label cosmetics app, ‘Face’ works along similar lines, allowing users of smartphones and tablets to try on different colours, shades and textures of lipsticks, eyeshadows, blushers and foundations. Rimmel London were one of the adopters of this technology, launching a new app using the platform this summer. Ikea have also made AR available to their customers to help them visualise Ikea furniture in their own houses.

In the logistics field, one major potential benefit of AR is achieving greater efficiency in the picking process. DHL have trialled the use of smart glasses and AR technologies in partnership with the software provider Ubimax, which specialises in industrial wearable computing solutions, and customer Ricoh. Using graphics to guide their staff and reduce picking errors, DHL reported 25% improvements in the picking process.

Another interesting application is real time translation. The app formerly known as ‘Word Lens’, which has been acquired by Google and incorporated into Google Translate, uses a smartphone’s camera to identify and translate text in real time on screen, displaying the translated words superimposed upon their original background. The technology was demonstrated on Google Glass before that project was discontinued.

Then, of course, there are AR games. Before Pokemon Go, there was Ingress, a popular ‘massively multiplayer’ location based sci-fi AR game which was developed by Google and provided the platform used by Pokemon Go. Whether these games should rightly be categorised as offering true AR is disputed, but, regardless, they are significant as they have brought the huge potential of AR to a wider audience. In another recent example, as part of its association with the world match play golf championships, Dell released an app that integrates with Foursquare, allowing users to play golf around the streets of the city of Austin, Texas.

Controversies

In terms of controversies, generally speaking, aside from the Pokemon Go app and Google Glass, AR devices and applications do not yet have sufficiently widespread use to prove particularly controversial.

The concerns that have been raised in connection with the technology class broadly fall into two categories—user safety and privacy. In terms of safety, in the UK the NSPCC raised concerns about the potential for Pokemon Go to compromise child safety and has issued guidance to parents on how to protect the safety of their children. There have also been numerous reported cases of users injuring themselves or putting themselves at severe risk while using the Pokemon Go app, including reports of:

  • children trespassing on the railway while hunting Pokemon
  • car crashes caused by drivers using the app
  • criminals using the game as a tool to provide opportunities for them to rob victims
  • users being shot having been mistaken for burglars.

Readers will note, however, that many of these risks arise from playing a location based game in public as opposed to AR technology per se.

In relation to privacy, the debate that was provoked by Google Glass, which incorporated functionality allowing users to take videos, gives a sense of the type of concerns that have been raised in connection with AR devices. At the core of this debate was that the video functionality of the product allowed users to video people in secret, which raised broader concerns about AR enabling people to harvest detailed information about other people. This is discussed later.

What functional and technical issues do AR developers have to grapple with?

Of course, the best route to understanding the functional and technical difficulties presented by the medium is to ask a developer—although in my experience this usually results in a lengthy discussion. At a high level some of the challenges I am aware of include:

  • enhancing the capabilities of AR devices to ‘see’ or map the word around them and therefore produce credible AR experiences
  • ensuring digital objects manifest the same visual phenomena as actual physical objects such as perspective, occlusion, the illusion of depth, eg ensuring that digital objects pass in and out of focus in a similar manner to the physical objects they neighbour
  • hardware limitations such as battery life and the computing power of holographic and graphics processing units
  • achieving an appropriate ‘form factor’ (size, shape and weight) for AR devices
  • ensuring that devices, software, and associated hardware and networks are secure, which is critical given the substantial volumes of data that will be processed about users
  • ensuring that wearable devices are able to capture successfully user inputs such as gestures or words to control their operation in an intuitive way (the new Hololens product is controlled by the user’s gestures and voice as well as a clicker device and Google Glass was controlled by touch and voice inputs).

Where AR is delivered in a mobile context, a further challenge is ensuring that the relevant digital content is triggered correctly on the basis of the sensor data of the mobile device (accelerometer, GPS, etc) and addressing the potential inaccuracies in the data generated by such sensors. There is also the challenge of achieving accurate location data within buildings where GPS functionality cannot be used. For completeness, I note that there are various different mechanisms to trigger AR content on mobile devices. The AR content may, for example, be ‘marker driven’—ie, there is a local trigger for the relevant digital content, such as a QR code or beacon device—or GPS driven.

How does projection mapping for AR work and what potential does this technique have in terms of enhancing advertising, books, games?

Projection mapping describes the technique of projecting video images onto physical objects, often with an irregular shape. This is achieved by using software which ‘maps’ the target object, which could be anything from a building to a person’s face, and controls a projector which projects the desired video or images onto that object. 

Again, there is debate as to whether this is a true form of AR technology or not, which depends on your accepted definition. Regardless of this, the results are often very impressive and there are many entertaining examples available on the internet. For example, fans of ping pong should check out the ‘Table Tennis Trainer’, and this example of projection onto the human body commissioned by Samsung is very vivid. 

In the field of advertising, the possibilities for projection mapping are great as any surface becomes a potential advertising board, including moving objects. One of the more common applications of projection mapping is to project large scale promotional videos onto the side of buildings. Advertisers have embraced this technology, creating high profile live installations in cities, as well as to form the basis of recorded adverts such as the latest Lexus advert as an alternative to CGI. 

The technology has also been used to illustrate pop up books with moving images. A pop up version of Macbeth produced in collaboration with the Royal Shakespeare Company gives a feel for what might be achieved. 

We are also seeing the first interactive games harnessing projection mapping technologies, for example, see this simple game developed by B-Reel London, or this prototype by Unity. What these instances illustrate is the great potential for video games played in three dimensional environments.

Of course, the effect of projection mapping could equally be achieved by simulating projections using AR headgear, and this is something that we may see more of in the coming years. 

What protection is AR technology currently afforded (ie, patents, copyrights, publicity rights)?

The constituents of AR technology—hardware, software, networks, digital content—are already well-known to the law of England and Wales. These constituents and their combination in AR offerings will be protected in much the same way as other more established technology classes.

As with smartphone technologies, we are seeing something of an arms race between the major technology companies in connection with AR technologies. Since the late-2000s, the established tech-sector giants have been acquiring and investing heavily in AR technologies and startups, with the strength of patent portfolios a particular area of focus.

A detailed survey of IP law in England and Wales as it applies to AR platforms and applications is beyond the scope of this Q&A. However, briefly, in overview: 

  • the software underlying the AR technology will be works protected by copyright and may also be protected by database rights and rights in confidential information—such software could also potentially be patentable in limited circumstances
  • the form and appearance of AR devices as well as their associated design documents may be protected by registered or unregistered design rights and copyright
  • AR hardware, and combinations of hardware and software, are potentially protectable by patents, as well as the largely obsolete semiconductor topography rights
  • content delivered by AR will be potentially protected by copyright (likely comprising a variety of different copyright works) and associated moral rights, design rights and trade mark rights and rights in passing off.

Since AR captures such a diverse range of solutions, in each case the analysis will depend upon the precise nature of the technology under consideration. Some of the really interesting questions around IP and AR technologies will arise as a result of the reproduction and modification of third party content processed by AR platforms and applications, and the permissibility of this.

How do you envisage the law/legal framework adapting to the impact of new AR technologies in areas such as privacy and negligence?

The widespread adoption of AR technologies will unquestionably present novel challenges to the legal system. However, it is difficult to predict with confidence how the law will evolve, as the societal impacts of this nascent technology class have only just started to manifest.

One of the principal areas in which we are likely to see legal developments in the wake of the mass adoption of AR, is the data privacy field. In the wake of the launch of the Google Glass product in 2014, the Senior Technology Office of the UK Information Commissioner issued a blog post, highlighting the importance of compliance with data protection law by wearables providers, noting media reports that the use of Google Glass had been banned in certain bars in San Francisco due to concerns the device might be used to secretly film people.

A potential application of AR that the larger technology companies have yet to fully embrace, likely due to privacy concerns, is the use of facial recognition software to enable AR devices to identify individuals in real time. It was recently reported that a new app called ‘Findface’ launched in Russia could identify strangers with a 70% success rate by harvesting publicly available profile information from a Russian social network. The significant societal implications of tools such as these are unquestionable and it would be surprising if this did not trigger further responses from regulators.

As AR platforms grow in importance, as a minimum, I would expect further guidance from privacy regulators as to how to remain within the law while using AR technologies commercially. Whether we will see any legislative developments in this area remains to be seen.

In relation to negligence, I have written elsewhere about the likelihood of claims arising as a result of injuries sustained by users of AR apps like Pokemon Go. How such injuries may be sustained and the degree of responsibility that providers could have for this (if any) will depend upon the nature of the device and applications involved.

It is likely that mobile and location based AR will provoke the first claims from users, as users are injured either through distraction or being taken to places of increased risk by the devices they are using. It will be very interesting to see to what degree it is determined that providers of the AR technology have a duty of care to their users, and how much users’ own responsibility for their personal safety is emphasised.

As an aside, I believe that the AR industry, and in particular app developers producing location based AR games and applications, would benefit from a voluntary code of practice to establish common standards that providers need to meet in connection with user safety.



This article was originally published on Lexis®PSL and the interview conducted by Alex Heshmaty. For more information on Lexis®PSL, please visit http://www.lexisnexis.co.uk/en-uk/products/lexis-psl.page.

News

4 April 2018

Browne Jacobson strengthens London commercial retail practice with double hire

Browne Jacobson has broadened its commercial retail offering with the appointments of commercial property partner Susan Voice and commercial solicitor Fiona McGurrin into its London office.

Read more

7 February 2015

Browne Jacobsons IP practice praised by leading independent directory

Browne Jacobson’s intellectual property practice has been named as one of the leading in the country in World Trademark Review’s (WTR) 1000 guide to trademark practices and practitioners.

Read more