In my previous post I published a survey of nearly 70 different devices that are in development or already available on the market that can be used to perform some form of augmented reality. A great deal of research went into this exercise and I learned a lot about the head mounted display market and technology while conducting it. In this post I will distill this information down to provide a better understanding of the essential elements of AR technology that are integral to enabling the ideal experience.
There is an increasing consensus that AR will be an on-demand experience rather than an always-on utility. There are head mounted devices on the market today that are geared toward “life logging” or recording one’s point of view experience with still and/or video then sharing that through social media, but this should not be confused with AR. Not unlike cell phones that are taken out of the pocket only when needed, head mounted displays will be donned when an AR task or experience is desired. When done with the task they will be put back into their case like we do with a good pair of sunglasses. Those who leave them on unnecessarily will be derided (as “glassholes” are), much as someone who unnecessarily wears their sunglasses when indoors.
While head mounted displays do need to look cool and be minimal in size, weight and intrusiveness, a certain degree of awkwardness will be acceptable as long as usage remains selective as described above. Though it would be nice to have HMDs that are indistinguishable from normal eyeglasses, this is not a realistic expectation of the technology any time soon. Such an expectation would be analogous to demanding a bike helmet that is indistinguishable from hair. As we’ve learned from the glassholes, always-on devices make other people uncomfortable.
More important than making the technology disappear is the experience the technology can offer us. Compromises in utility in the name of fashion is a sure recipe for failure. Contrary to the opinions of others who have written on this subject, AR HMDs should not be approached as “lifestyle” statements. It is the use cases that will drive adoption. There must be compelling reasons for people to don the devices. These use cases must provide value beyond mere novelty. For more on use cases see my previous blog entry here.
There are many technological elements that must be woven together to produce the ideal head mounted AR display for mainstream use. At this point, no one product has yet achieved this ideal, but it is apparent that several well-funded companies are racing toward this goal. In my assessment, the elements that will make for truly great AR experiences are as follows:
- Form Factor
Displays that resemble glasses are perhaps the least obtrusive form because people are used to both wearing eyeglasses and encountering others with this accoutrement on their faces. There is a limit to the size of something that can be worn as spectacles thus making them ideal for most consumer applications where bulk is undesirable. Visors such as the recently announced HoloLens from Microsoft may be better suited for active pursuits such as gaming or sports. A form factor such as the Daqri Smart Helmet would only be appropriate for industrial or military applications.
Design is important in all personal technology and has been the key to success for companies such as Apple. That said, form must follow function and comfort. Designs that integrate the myriad sensors and processors that go into HMDs in a way that is stylish, attractive and has that “cool factor” is most likely to succeed with consumers. Gamers and industrial users will be less style conscious because they won’t be seen in public and therefore more tolerant of designs that are bulkier or geekier — as long as they’re comfortable to wear.
Part of the comfort factor is surely the weight of the unit. Users must be able to wear the device for an extended period of time without excess pressure on the nose bridge, ears or neck.
The easiest way to make the HMD smaller, lighter and better suited to a stylistic design would be to offload as many of the components as possible onto a tethered control unit. Epson has taken this approach with their Moverio product. Perhaps the greatest advantage of doing so would be the capacity to accommodate a larger power cell to eliminate battery life as a limitation on the AR experience. Other components such as CPU, GPU, memory, storage, SD card slot, radio circuitry (BlueTooth, WiFi, NFC) and control buttons can also be integrated into the control unit without compromising functionality. The trade-off of course is having a cable running from the back of the unit to your pocket, but this does not seem to be a style barrier for the millions who sport earbuds wired to their smart phones and music players. A compromise to this approach would be to leverage a BlueTooth connected control unit but this would move some of the components such as battery, radio circuitry and some processors back to the HMD as well as adding another degree of latency to the ever important synchronization of the augmented information overlay. Though the ideal HMD should be entirely self contained, sticking to my mantra of form following function and comfort I believe that a consumer solution that includes a tethered control unit is acceptable until battery technology has evolved to a state of greater power in a smaller, lighter cell. Another rational scenario is for smartphone makers to build AR support into their units which then can double as a tethered control unit (wired or wireless) when an AR experience is desired. This makes sense because smartphones already contain many of the components needed by an AR HMD and much redundancy between the two may be eliminated. Technologists agree that the HMD will replace the smartphone altogether for many people. Look for phone makers who produce their own microprocessors such as Samsung to test this approach.
- Human Interface
As a new media communications paradigm, augmented reality will inevitable breed new means of interacting with technology. This is sure to evolve over time but it is inevitable that we will at some point be interacting with technology in ways we have not even conceived of yet.
One of the great promises of augmented reality is the enablement of hands free task execution while receiving information or instruction through the eyes and ears. In this new paradigm, one needs a way to send commands to the OS to navigate the menus and control the device functions. When the hands are occupied, the best way to do so is with voice command. Siri and Google Now are proof-positive that voice recognition has matured to the point of being reliable enough for AR interaction. The ideal AR HMD system must be voice command enabled.
Of course there are times and places when vocalized commands would be impractical or socially awkward. Sitting amidst a roaring crowd at a sporting event is one example. An alternative interface to commanding an AR system is gesture control. I will be covering gesture control more in depth in a future post, but basically there are two ways to accomplish this: wearable motion sensor devices such as Ring and depth sensing computer vision. The former option would be peripheral hardware that communicates with the AR system via BlueTooth while the latter would be integrated into the HMD itself, as found in the ODG R-6. Both types of controls have their advantages. Depth sensing sees the user’s hands and can effectively allow the user to touch and swipe virtual controls in their field of view. Motion sensors are handy when it is impractical to wave ones arms in front of the depth sensors such as at the aforementioned sporting event. Devices like the Ring allow for more subtle forms of gesture. Motion sensors’ six degrees of freedom also allow for more precise gestures such as writing in air. An AR HMD should support both types of gesture control.
- Physical Controls
While voice and gestures will allow one to interact with their AR experiences, there are other forms of control to consider. There are basic functions such as power, volume and left/right/up/down scrolling and selecting that should be available on the tethered controller or HMD itself that can be used for configuration or in place of gestures when required. These may be in the form of buttons or a touch sensor.
- Brain Waves
Brain waves? Yes, brain waves! Daqri just bought the EEG-tracking headband company, Melon. According to DAQRI CEO Brian Mullins, “The EEG space has immediate potential to enhance 4D wearables with safety features, as well as long term potential to create a game-changing brain-computer interface that will allow you to control 4D interfaces and objects in the real world.” The advent of controlling technology with our thoughts has game changing possibilities.
There are several types of sensors that should be present on an AR HMD device to deliver rich experiences. These sensors can all be found in existing smart phones and have been sufficiently miniaturized to fit in an HMD. You can find deeper explanations for many of these sensors on this previous blog entry I wrote.
Orientation tracking allows the device to know where it is in space so that AR applications can react to movement, attitude, direction and location using accelerometer, gyroscope, compass and GPS, respectively.
- Ambient Light
When overlaying images and information over one’s view, the amount of background light affects the viewability of that information. An ambient light sensor is necessary to measure that background light and brighten the overlays when there is more light and dim them when there is less.
Microphones are not only used for voice commands as described above, but also for telephonic communication. One of the more intriguing AR use cases is remote assistance where the video camera on the HMD is used to provide the user’s point of view to an off-site expert who can then guide the user through a task. The microphone is then used for voice communications between the user and expert. The HMD microphone should be paired with noise canceling circuitry.
- Air Pressure
Altimeter/barometer sensors measure air pressure. In addition to informing applications for navigation, skiing and cycling, it can aid in GPS location locking speed and accuracy. This is a must have sensor for AR HMDs like the Recon Jet that are intended for outdoor adventure but may not be essential for most units.
There are three different types of cameras that are needed on HMDs: still, video and depth.
- Still cameras
Still cameras don’t necessarily have a key role in rendering AR experiences but one could think of different applications that would take advantage of this feature, especially those that incorporate a social sharing element. 5 megapixels is a sufficient resolution for this.
In addition to the remote assistance scenario, video cameras provide a means to record hands-free what one is doing and seeing. A resolution of 720P is sufficiently high definition to cover all use cases without hogging too much power and storage, but 1080P would be preferable.
Depth sensing cameras are found in the latest generation of AR devices hitting the market such as Magic Leap. In addition to being able to read hand gestures as I have already discussed, it has a much more significant role in providing a true AR experience. The richest AR applications will recognize surfaces and objects in three dimensions and overlay information and images that take into account the context of one’s surroundings. There are different types of depth sensors but the best option for mobile AR is known as a Time of Flight (TOF) camera which uses scannerless LIDAR consisting of an infrared LED or laser to pulse-illuminate the area and an optical lens to focus the reflected light onto an image sensor at speeds up to 100 Hz. Logic circuits then interpret the reflected light as depth.
- Still cameras
The visual display is perhaps the most important component of rendering AR experiences and there are several different approaches being taken by the HMD makers. These approaches are far too complex to get into the technological particulars of in this posting but there are certain qualities of display that should be understood.
While several HMDs are of monocular design, a rich and engaging AR experience will require a binocular view through which stereoscopic images may be projected to effect 3D or holographic images.
- Field of View
FOV describes the image size relative to the point of view of the eye. Humans have nearly a 180 degree horizontal and 135 degree vertical FOV including peripheral sight. The closer to this FOV the HMD can reproduce the more immersive the AR experience will be. Meta has one of the better FOVs today at 35 degrees. FOV is a factor of the resolution of the display (i.e. SVGA is 800×600 pixels ) and the optics that bring the image to the eye. This will be an area of improvement that all makers will be focusing on for competitive advantage.
Luminance of the display is an important consideration of performance and usability under different lighting conditions. Luminance is measured in candela per square meter (cd/m2) or sometimes in nits. The SeeThrough from Laster Technologies boasts 5200 cd/m2. Bright sunny conditions pose challenges for see-through displays. A photochromic lens (automatic darkening) would be an ideal means of mitigating this problem while some makers are taking an approach that includes a supplementary snap-in shaded lens approach.
- Color Depth
The number of colors producible by the display is also of importance to the experience and is known as color depth. The human eye can discriminate up to ten million colors. The Epson Moverio is 24-bit (16,777,216 colors) while the Sony SmartEyeglass Developer Edition is only 8-bit monochrome green.
- Refresh Rate
Also known as frame rate, this describes the frequency at which an imaging device produces unique consecutive images to the eye and is measured in frames per second (FPS). A frequency lower than about 12 FPS is perceived by humans to be individual images while faster rates create the illusion of motion. 24 is the current video standard which would be the expectation for HMD display refresh.
- Eyeglass Compatibility
Compatibility with prescription eyeglasses which is a must for mass adoption of AR HMDs.
AR is not about visualization alone — sound can be a very important component to the experience.
- Bone Conduction
Google Glass utilizes integrated bone conduction transducers. This type of speaker does not have a moving membrane like traditional speakers, rather a small metal rod is wrapped with a voice coil. When current is pulsed through the coil, the magnetic field causes a piece of metal to expand and contract. When pressed against the jaw or skull it turns bone into a speaker. The effect is quality sound that seems to be coming from within one’s head which others cannot hear, yet since the ears are not covered the user is not isolated from ambient sounds and therefore not vulnerable to the dangers of traditional earphones.
Earbuds would also be an acceptable means of transmitting sound to the user and is a necessary capability for when the unit is to be used in loud environments.
- Bone Conduction
- Radio Communication
The AR HMD needs to be able to communicate with the world in many ways. To do so, it must be wired with different radio types. These are the same radio circuits found in the latest smartphones.
WiFi is key to connecting the HMD to networks. Many AR apps integrate with APIs that utilize data services located in the cloud. The fastest, most efficient and low cost way to access this data is through a WiFi network when available.
Bluetooth is the ideal protocol for connecting the HMD to peripheral devices. This might include phones, tablets, laptops, keyboards, mice, gesture control devices, automobile entertainment systems and even the Internet of Things. The latest standards version uses less energy and has a greater range than its predecessors.
- Near Field Communications
NFC technology enables devices to establish radio communication with each other by touching them together or bringing them into close proximity. It is being deployed for many novel applications such as commerce, initiation of connections of other radio technologies, social networking, identity authentication, triggering applications and commands, as well as gaming. AR HMDs can take advantage of many of these same use cases, but it is unclear whether it would be practical to expect to have the HMD come in close proximity to other devices while on the head.
Cellular networks are key to rounding out the HMD’s role as a communications device. When not within range of a WiFi network, a cellular network is used for the transmission of data. The HMD should employ a fourth generation (4G) cellular technology in order to do so at reasonably high speeds. Equally important is enabling the HMD to be used for telephony, such as in the remote assistance use case mentioned earlier in this post.
ANT is an extremely power efficient wireless communication technology that allows connection to various ANT+ sport, fitness and health devices such as heart rate sensors, fitness equipment, cycling products and more. In the future it will also encompass home automation control of lighting, temperature and door lock functions. While ANT may not be of relevance to most AR applications, inclusion of it by HMD makers will provide a competitive advantage by appealing to users who want experiences in these areas.
Of course supporting all of these essential features discussed so far requires sophisticated and powerful processing capability — especially when running them simultaneously and in concert with one another. Size, power consumption and heat radiance are also important considerations. The HMDs on the market (or near-market) today employ systems on a chip (SoC) technology which is a circuit that integrates all components of a computer or other electronic system into a single chip. Most modern mobile electronics employ SoCs for their low power consumption qualities. HMD SoC should have the following qualities:
- A CPU or microprocessor with a minimum of two cores. The latest generation of smart phones have eight cores and sophisticated HMDs will also need this amount of power for the demands that AR will place upon them.
- A GPU (graphics processing unit) is needed to process and display 3d images with minimal latency. Originally developed to support the demands of gaming, GPUs are indispensable for state of the art AR HMDs.
- RAM is also built into the SoC. Todays units have 1 or 2 GB of RAM to handle temporary storage of data. Look for this to soon go up to 4 GB for the state of the art HMDs.
- The SoC should have built in capability to accommodate flash memory as well as extensible memory in the form of user added SD cards.
- External interfaces to industry standards such as USB, Ethernet, etc. must be present on the SoC.
- The bus is a communication system that transfers data between these aforementioned components on the SoC. The most common bus architecture on mobile devices today was developed by British company ARM Holdings. Arm’s is a RISC-based design approach which reduces costs, heat and power use. The Qualcomm Snapdragon series is an example of an ARM chip that might be used in HMD displays today.
Power is a very important consideration for all mobile devices. A balance must be struck between rich features, battery size, battery duration and charging time. Battery technology seems to be the toughest problem for engineers to crack. An HMD is worthless if its power has been drained when you want to experience AR. Yet the battery from my Samsung Galaxy S4 weighs in at 44 grams which would add significant heft to an apparatus resting on ones nose and ears. Earlier in this post I state that one way to mitigate this is to include a tethered pocket-size controller in which a heavier, more powerful battery may be seated. There may be other options as well though, such as hot swappable batteries. The HMD could contain a capacitor that stores 60 seconds of power giving the user enough time to pull a new cell out of their pocket and snap it into place in an easy to access locale.
- Battery Type
Lithium-ion batteries are the most popular types of rechargeable batteries for portable electronics because they have a high energy density, no memory effect, and only a slow loss of charge when not in use. This remains the best bet for AR HMDs for the foreseeable future.
- Battery Capacity
Battery strength is measured in milliampere-hours (MaH) which represents one-thousandth of an ampere-hour, which in turn is equal to the charge transferred by a steady current of one ampere flowing for one hour. Higher MaH ratings translate to longer use. Something in the neighborhood of 2600 MaH should provide acceptable battery life.
Inductive charging technology uses an electromagnetic field to wirelessly recharge batteries. This eliminates wires and connector compatibility issues between device brands but is ultimately a slower and less efficient means of charging. However the rise of the Qi standard may lead to the presence of universal charging pads embedded in everything from Starbucks tabletops to automobile consoles making for ubiquitous access to charging. I think Qi has great potential and building wireless charging into AR HMD devices will provide competitive advantage.
- Battery Type
- Operating System
Though Apple devices are inescapably popular, the closed nature of iOS rules it out as an AR platform for independent makers of HMD devices. The Windows platform is also closed but the development environment support for this operating system is so extensive and robust that it must be considered a viable option. That said, the open platform of Android is ultimately the best operating system to build upon. Many current HMD makers have created a forked version of Android to suit the specific needs of their devices.
Just because the HMD runs on a flavor of Android does not mean that it cannot be used in conjunction with non-Android devices. AR HMD devices must be able to communicate with smart devices running other operating systems, especially in consideration of the scenario I posit above where the HMD is tethered to an AR-enabled smart device.
A successful AR HMD will entice developers to build applications, games and experiences that run on it. The more content that is created for it the greater the adoption curve, which translates into sales and profits. A rich SDK makes the development effort easy and flexible. An HMD maker should put a great deal of resources into maturing the SDK in short order.
The ultimate goal for any HMD maker is for it to survive the Darwinian forces that inevitably winnow down the technology choices to two or three winning platforms. Once this happens the barrier to entry becomes overwhelmingly difficult for newcomers. In addition to building a powerful SDK, the successful HMD maker must cultivate a rich ecosystem of partners in the hardware, software, commercial and retail channels. These partnerships beget new partnerships as the technology gains critical mass. HMD makers should look to follow the example of the wireless industry in this regard.
- Operating System
Cost will be a key component to consumer adoption. This is one reason why some believe that AR will first be adopted by industrial and manufacturing sectors who can better absorb the high costs of early adoption. Their investment will fund the innovation that will ultimately bring the price down for consumers. The amount that the HMD maker can charge consumers will be a function of the perceived value that the technology has. Today, consumers are willing to shell out six or seven hundred dollars for a state of the art smart phone. But a phone is something that many people use every hour of every waking day. Until HMDs can replace smart phones, they will be used considerably less. On the other hand, if the use cases are compelling enough the value proposition may very well overcome this disparity in perceived value as compared to the phone. What the cell phone model has taught us though, is that consumers will indeed be willing to make such an expensive purchase if they can make easy to afford payments. Since many HMDs will contain cellular network connectivity, perhaps the primary sales channel for them will be through the service providers AT&T, Verizon, T-mobile and Sprint. This, in turn, may serve to hasten the HMD’s replacement of the smartphone for many people.