The Holy Grail of augmented reality technology is a wearable medium by which information and 3D images can be integrated into one’s view of the real world where it is in registration with the objects and surroundings being viewed and dynamically adapting to those surroundings in real time both positionally and informationally. There are a number of companies (whom I will cover in future posts) today who are either accomplishing this in a limited fashion or have something close to this ideal in development. While implementing this ideal into spectacles that are attractive, lightweight and unobtrusive is the ideal form factor for mobile AR, it is not the only medium for AR experiences. In this post I aim to identify the many means by which AR experiences can be implemented visually. In subsequent posts I will cover the other technological aspects that go into rendering AR experiences which complement this visual presentation.
Though a rudimentary interpretation of the definition of augmented reality, perhaps the most familiar mode of AR is the superimposition of graphics on a video screen. In sports this was first manifest as the First and Ten Line by Sportsvision in 1998 and has since grown to encompass dozens of ways of integrating information into the fields of play in almost all televised sports.
The same concept can be found in cars with in-dash video feeds from rear-facing cameras where the trajectory of the car is delineated with colored lines and distance cues to ease the task of parking and improve safety when backing up. These examples may not be regarded as augmented reality by some because they are not a first-person view.
Mixed Reality Video Display
A modality of AR that is entirely composed of a video image is exemplified by a concept that includes a video display inside non-transparent goggles that combines augmented CG information integrated into a view of the real world provided by a head mounted camera. The user sees nothing other than what is presented to their eyes through the video feed. Though this technology was accomplished years ago, the prospect of navigating the world entirely by video feed may be disorienting and cause motion sickness due to the inherent signal processing delay and therefore not practical for commercial implementations.
Handheld Device Screens
Another form of all-video AR can be found in apps on modern mobile smart devices that have built-in cameras and view screens. One points their device camera at a place or thing and the app overlays information or imagery about it on the view screen. Scores of apps have been developed for navigation, tourism, entertainment, procedural tasks and advertising that utilize this approach. This technology has matured to the point where it has become very useful and relatively common but has shortcomings that greatly curtail its usefulness: at least one hand must be holding the device which limits the hands-on tasks that can be accomplished with it and the small viewing area on phone screens limit the amount of information that can be displayed to the user.
Heads-Up Displays (HUD)
Based on the limitations that handheld AR imposes on the user, it has become clear that hands free solutions must be developed in order for the true potential of AR to be realized. As detailed in my post 04: Origins and Evolution, heads-up displays have been in use in aeronautics since the late 1950’s. Projection of information such as speed, altitude and attitude on the pilot’s windscreen allow them to keep their eyes on the horizon rather than down at their instrument panel. The same approach has been attempted to varying degrees of success in automobiles over the last 30 years to display driving speed on the windshield and is now being extended to navigation applications.
The military has also long made use of near-eye microdisplays to deliver situational awareness to ground troops and pilots alike. This is essentially a tiny screen or optical eyepiece for viewing a display placed in front of or near the periphery of the eye that the wearer must shift their gaze upon in order to view.
It involves color-producing pixels on a tiny rectangular substrate that are then magnified and their light directed toward the eye using optics.
There are several different microdisplay technologies with potential for implementation on wearable devices: LCD, LCOS, DLP, OLED and Laser Beam steering. Each option offers different advantages and disadvantages in weight, resolution, brightness, power consumption and cost.[Karl Guttag] Though wearable microdisplays can be considered a form of AR, they only display information near the eye which falls short of the goal to overlay 3D information and images stabilized on a specific position on a moving real world image. The disadvantage of this approach compared to others is that the microdisplay is not translucent and therefore blocks from one’s field of vision anything behind it. Because of this it must be placed slightly outside of one’s field of vision thus requiring the wearer to take their gaze off of the task at hand making it impractical to use while operating a vehicle or even walking in many circumstances.
Projected Image Overlay
With projected image overlay we finally introduce a technology capable of achieving the ideal AR experience I described at the beginning of this post.
This is a method of using optical waveguides to project or reflect the augmented information from a microdisplay onto clear or tinted eyeglass lenses directly in front of the eye. The light is then reflected back into the eye allowing the image to be effectively superimposed over the view of the real world. There are several different approaches to accomplishing this, each with its own advantages and disadvantages related to luminosity, obtrusiveness and size and weight of the apparatus. The advantage this approach has over near-eye microdisplays is that the image is see-through and does not block one’s field of view. One disadvantage is that brighter lighting conditions are likely to wash out the image thus rendering it less effective. Another challenge in perfecting this approach lies in the fact that the image projected on the lens lies on a separate focal plane. Implementing this in such a way that does not require the wearer to change focus in order to view the AR content is necessary to prevent eye fatigue.
Dual Focus Contact Lens
This technology allows the wearer to focus on two things at once – both the information projected onto the glasses’ lenses and the more distant view that can be seen through them. The ability to focus the near-eye image is achieved by embedding optical elements inside the contact lens that are so small that they do not interfere with the wearer’s normal vision. The lenses have two different filters. The central part of each lens sends light from the display towards the middle of the pupil, while the outer part sends light from the surrounding environment to the pupil’s rim. The retina receives each image in focus, at the same time. This technology builds on projected image overlay technology but solves the problem of focus. It’s doubtful that a contact lens based technology would ever have mass market appeal but the military has certainly taken notice as evidenced by DARPA’s 2012 investment.
Near-Eye Light Field Displays
The latest entrant in augmented reality head mounted display technology has been brought to light (pun intended) by the major investment in a company called Magic Leap by the likes of Google, Qualcomm, Kleiner Perkins, Andreessen Horowitz, and Obvious Ventures among others. Though this company’s offering is still quite vague, it has made claims of “cinematic augmented reality” which its patents indicate is being accomplished through the use of dynamic light-field display optical hardware. The patent describes how such a device that it calls “waveguide reflector array projector,” would operate. The display would be made up of an array of many small curved mirrors; light would be delivered to that array via optical fiber, and each of the tiny elements would reflect some of that light to create the light field for a particular point in 3-D space. The array could be semi-transparent or reflected off see-through lenses via wave guide to allow a person to see the real world at the same time. Multiple layers of such tiny mirrors would allow the display to produce the illusion of virtual objects at different distances.[Technology Review]
While I previously described the ideal augmented reality experience as involving a head mounted display, this does not mean that there aren’t other novel means of achieving AR. Spatial AR uses a projector to overlay augmented content onto actual surfaces in the physical world. In fact, at the Wired Magazine Next Fest held in Chicago back in 2005, I had the opportunity to try a device that sensed the location of the veins and arteries in my arm then project them onto to the surface of my skin. This technology allows a medical worker to identify locations of veins for quick and effective injections or IV taps.
Spatial AR has been proven in other interesting applications as well. In my opinion, an interactive keyboard that is projected onto a surface meets the definition of AR. But much more intriguing is the concept of a “transparent cockpit” that projects a view of the surroundings of a car onto the interior surfaces allowing the driver to effectively see through the car as though it were made of glass.
Though far from being market-ready, the concept has been proven. Here is an intriguing video made by the researchers.