In previous entries I have defined AR and the technological context in which it has evolved, but a further understanding will arise when one is exposed to the myriad use cases for it. Below I offer examples in a number of different categories, some of which already exist or are in development, while others are speculative in nature. What other use cases can you come up with? Be sure to let me know in the comments.
AR will enhance the performance of critical tasks by enforcing methodical execution of assembly or other operations by providing visual guidance, tracking, 3D animation and instruction.
AR technology will allow a technician to execute a guided process for highly critical tasks such as assembling a jet engine. In today’s world he would carefully consult a manual (paper or computer based) for each step then execute it accordingly going back and forth between each step. Using AR, the technician can be hands on with the objective while receiving visual cues to find the right position and use the correct tools and audio to execute the necessary steps in the right order. It can provide information at the point and place of need. It can even adjust to the current context: a technician can start by himself and request support from an Augmented Reality application when he cannot continue based on his own knowledge.[Juergen Lumera]
AR could be used to improve social interaction by displaying certain information about the people one encounters based on facial recognition technology. Like all social media, one would need complete control over what sort of information is exposed to others based on how they classify their relationships. Faces one encounters could be read by the camera on a head mounted display then sent to a trusted social media service provider through which the data is controlled. Peoples names, their connections to the viewer’s social graph, professions, affiliations, “likes”, recent posts, etc. Similarly, such applications could be used in social situations where people do not know one another such as singles bars/events or in professional venues such as conferences and trade shows where perhaps different types of data would be made visible.
Social media can be combined with geospatial location data to affix virtual tags to physical places that are made visible to those in one’s social graph. Restaurant reviews, photos and video, purchases, etc. are activated when in the vicinity of friends’ tags and viewed through the device screen.
Public Space Gaming
Innovative AR games bring people into public spaces to “gamify” the exploration of their surroundings, learn about points of interest, or collaborate with friends and strangers to achieve common competitive goals.
Use of Gesture Recognition
Games previously confined to specific venues are played anywhere with the aid of gesture recognition. Virtual paintball can be played in a mall and participants can use their fingers as guns to shoot opponents with virtual globs of paint visible only to the games participants that are then wiped away with the reset of a new game.
Incorporation of Surroundings
Video games are removed from the confines of the screen. Players of a game such as Asteroids in an urban setting might see 3D boulders hurtling toward them from the sky and bouncing off buildings, buses and benches then breaking into ever smaller pieces when virtually blasted.
Transformation of Traditional Games
Tabletop games will be revolutionized. A tabletop can become a gridiron with goalposts on which kids can play with a physical paper football. Incorporation of gesture recognition could enable a game of virtual chess where there is no physical board or pieces. The game of pool can be played with no physical balls or cues.
Augmented reality has been a part of watching sports on TV since 1998 in the form of the 1st and TEN® marker which projects a yellow line on the field where the ball must pass to achieve a first down. Since then, scores of augmentations have been introduced throughout most televised sports including pitch trajectory trackers in baseball, driver trackers in motorsports, golf shot trackers, pool lane swimmer markers and of course virtual ad placements.
Enhancing Live Events
The wearing of AR glasses will transform sporting events for fanatics by identifying players on the field, displaying their statistics and information about their career. Graphical elements like those used in TV broadcasts (i.e. the 1st and TEN) may be superimposed on the event in real time. A social media feed can provide opinions and commentary by other fans in attendance while voice recognition allows one to share their own without having to take their eyes off the game to type on a device. All such options will be configurable to ensure the user is only getting the information they want when they want it so as not to overwhelm the enjoyment of being present.
AR will introduce multi-media content into the quaint tactile pleasure of paper media to extend its capability and appeal. This represents an opportunity for publishers of books, magazines and newspapers to reinvent themselves in the digital age without having to physically change their medium.
Magazines will be able to charge a premium for ads that have AR content tied to them. Taking the concept a step further the ad can be dynamically customized based on the profile of the person viewing it.
Enhancing Imaginative Play
AR will enhance how kids interact with such toys as Barbie dolls or Hotwheel cars by providing contextual imagery that was previously left to the imagination. A child zooming a car around on the floor might choose to overlay city streets, highways or racetracks to guide it through thus making the experience more visceral.
Augmented Reality has already found its way into museums in several different forms of implementation.
An exhibit called “Ultimate Dinosaurs” has appeared at museums such as the Royal Ontario Museum in Toronto and Cincinnati Museum Center that uses augmented reality to add flesh to the bones of dinosaurs and lets them move around. An app that can be installed on attendees’ smart devices makes beasts pop out of markers around the exhibit, including on the floor; in others iPads provided by the museum turn fossils into fleshed-out creatures. Along the walls are animated projections of dinosaurs that also are interactive. With the help of a Kinect 3-D camera, their eyes follow your every move.[Smithsonian]
In The Getty Museum in Los Angeles an exhibit titled “Life of Art” enables visitors to use iPads to explore in much more detail–and even rotate–classic historical objects from its permanent collection–a 17th century lidded porcelain bowl from Asia, for instance, and an 18th century French armchair.[Smithsonian]
The Science Museum in London’s exhibit on the Making of the Modern World is an educational experience about how technologies have propelled human history forward. The companion Science Stories app features a three-dimensional avatar of “Top Gear” host James May as a virtual tour guide. The visitor points their smartphone camera at a “marker” that the app recognizes as an area on which to project the avatar. The overlay of Mr. May appears on the screen as a 3-D photo model, walking and talking just as the real man would if he were at the museum with you. As the smartphone’s view of the marker changes, the avatar will adjust its orientation to the new view.[Wired]
Car manufacturers and several aftermarket vendors are now offering heads-up navigation systems that allow the driver to keep his eyes on the road while receiving route, road and landmark information both visually and audibly. The Cyber Navi system from Pioneer has a small screen in the place of a car’s sun visor for the heads-up display. A laser projects driving instructions and maps appear to float above the road about three meters away. A camera mounted near the rear view mirror constantly scans the road ahead and feeds video to a traditional LCD screen mounted on the dashboard. The system overlays the video with graphics, using object recognition to detect and highlight the nearest vehicle ahead as well as upcoming street signs and traffic stops. The dashboard screen also points out landmarks such as restaurants and gas stations, highlighting buildings or adding large icons that float in the air. The system also shares road condition data with other drivers using the same system.[Computerworld]
AR apps can be used to obtain information in a simple and direct way about places being visiting. They use a smart device’s camera display to reveal shops, restaurants, and points of interest as virtual signs overlaid on the buildings within view. From there one can browse for information by category and obtain real-time information about what they can find, where they can find it and how to get there.
Apps have been developed that allow people to see historic sites as they once appeared. These may be ancient sites such as the Coliseum in Rome or more recent history such as the New York skyline with the World Trade Center buildings intact. Pointing a camera viewer at the site adds 3D content with a perspective that changes depending on where the viewer is standing. This may be used for educational or tourism purposes.
Imagine visiting Gettysburg and firing up an app that displays lifelike 3D characters acting out a scene from the Civil War on the actual hallowed ground where it took place. An audio voice over can supply the narrative and additional contextual information can be raised through the touch interface to learn more about the event, the place and the people.
AR is changing the way we travel and communicate through real time translation. Apps exist that allow you to point the device’s camera at a short phrase of text such as a sign or headline and the view the translation on screen overlaid on the original text. The aural version of this concept translates spoken text to the earphones of the wearer. As the technology matures it will break down language barriers between people when traveling or conducting business.
Medical and Healthcare
Using an iPad, surgeons in Bremen, Germany, were able to create a virtual 3D pre-op plan for one patient’s procedure. The surgeons took a picture of the patient’s liver with the iPad’s camera. The app then constructs an augmented reality overlay of the liver, showing the physicians where essential structures such as tumors and blood vessels lie. The AR overlay is built from scans of the organ that were performed prior to the surgery. Having the actual layout of the organ visible before beginning allows surgeons to avoid potential complications and could result in more efficient operations.[Apple Insider] The maturation of wearable technology (i.e. glasses) has the potential to spark a major leap in surgical technology in the near future.
AR apps can be used by laypersons to treat injuries in places where there may not be professional care available. Such apps may take two different forms: Standard procedures like dressing a wound or setting a bone could be delivered via step by step audio feed with visual assist. This scenario would be viable in remote locations where there is no cell service such as at sea, in the wilderness or the desert. The second approach involves the engagement of an offsite medical professional who sees a video feed from the glasses of the person treating the subject then provides live audio instruction while working a telestrator to deliver visual cues. This could be used in the battlefield or in civilian situations such as disasters or accident scenes.
Seeing Eye for the Blind
The definition of augmented reality stretches beyond the realm of sight to enhance the ability for blind people to perceive their environs. A 3D camera system can turn visual data into a set of audio instructions that are transmitted to the wearer via a wireless headset. This technology can be used in conjunction with a cane or guide dog by communicating information beyond what those implements can provide such as distinguishing a door from a wall, which side of the door the handle is on and even which direction the door opens.
There are many ways in which AR can assist the human mind. One such use case is to help those who have a compromised memory from Alzheimer’s or Traumatic Brain Injury. By combining AR with facial recognition technology, the identity and relationship of people they encounter can be overlaid on their view. This will help them to function more normally and mitigate some of the fear and confusion these conditions bring about.
Augmented reality apps can be used to project factual information onto physical features to provide the user with knowledge, context and understanding. A great example of this is an app that allows you point a device at the night sky and it displays the names of the celestial bodies in sight such as planets and stars, draws lines to connect stars into constellations and facilitates clicking through to more in depth information about the objects. Similarly, AR could be used to teach anatomy by pointing a device at another person or one’s self and “seeing” through the skin to reveal 3D images and details about internal organs, musculature and the skeleton.
Text Book Supplementation
Many believe that paper form books will continue to have a place in the classroom for the foreseeable future. The text book’s evolution will be driven by AR which will supplement the flat page with images, video and 3D interactivity. Imagine a high school chemistry book that uses the photo of a molecule to trigger an AR experience where a 3D model of the molecule floats above the book where the student can use gestures to rotate it in any direction or zoom in on it from any angle to examine the bonds.
Helicopters are often employed to assist when pursuing a perpetrator or performing a search. When a helicopter is dispatched, maps are used by the tactical flight officers to navigate to a destination or to communicate a specific address to ground forces. Augmented reality technology has been developed that overlays routes, road names and addresses over the live video feed from the aircraft’s high definition camera. This is the same feed that displays infrared images which are used to identify the presence of persons on the ground. The integration of GPS AR into a single view is a powerful tool that brings tremendous value to law enforcement. As this technology evolves it will also be integrated into drones and have the ability to tag law enforcement personnel in the view to distinguish them from the subject.
Firefighters using an AR display can see directions projected in their field of view to help them search a burning building for people or find their way out when vision is obstructed by smoke or their path is cut off by flames or debris. They could be further assisted by personnel outside of the structure or at a command center who can see their point of view through a helmet mounted camera and help with emergency medical situations or provide team oriented tactical firefighting instructions. Additional sensors can detect and project surface temperatures to indicate hotspots.
When shopping for furniture in a showroom it can be difficult to envision what a piece would look like in your own home. AR apps can display furniture items in situ using augmented reality. By pointing the camera at the target location, different pieces in different colors can be visualized in different positions at scale by viewing the scene through a device display. Selling furniture online is generally a difficult proposition, but this AR advent allows vendors to offer many more options than can be displayed in a showroom and allows buyers to “try on” furniture before they buy it.
Buying art for one’s home is a highly personal experience. Everyone has their own tastes and every space in every room has its own characteristics that demand different sizes, shapes, colors and styles. AR apps can be used to browse a vendor’s collection of art by a number of different criteria. Point a smart device at the space where the art must be hung and the app allows one to visualize the different options on their own wall. Once a piece has been selected one can then try a number of different framing options before purchasing it and having it shipped.
Augmented reality enables people to virtually try on clothes in a store or at home. The technology helps shoppers to visualize how different sizes and models of garments compliment their unique shape as they would in a real-life fitting room, but without the need to physically try on the clothes. A shopper can rapidly try different colors and combinations or try on items that may not be stocked in the particular store they are visiting.
Shopping apps take a photo of a found product or barcode, identify it then overlay information and reviews about it on-screen and provide linkage to make the purchase. The term “showrooming” was coined to describe the practice of using brick and mortar stores to physically interact with and compare products then use a smart device to find the best price and purchase online. While this has empowered the bargain shopper, it has forced strategy shifts by retailers who must closely monitor online pricing to maintain parity and also offer on demand price matching. The evolution of this will be seen when AR glasses become ubiquitous: The smart device GPS recognizes that you are entering a store and opens the shopping app. As you walk down the aisle and stop to look at products, the app automatically looks them up and informs you if you are getting the best price.
Marketing & Advertising
Scores of advertising campaigns have incorporated novel augmented reality contrivances to get brands and products noticed. Magazine ads that trigger 3D images of products such as cars that can be rotated or zoomed into and even peered into from different angles. Billboards that come to life with lithe models on whom one can try different lingerie. Storefront windows that generate afterhours sales by capturing the attention of passersby with engaging content that showcases products whose purchase is facilitated within the app.
AR apps are being created by product manufacturers like Lego to provide previews of what the finished product looks like. The app recognizes images from the package, catalog or website to display a 3D model that can be rotated in space and viewed from different angles. Beyond toy models, this concept can be applied to other products that require assembly or preparation such as furniture or even food.
When the consumer interacts with a brand in the real world it becomes the ultimate in engaging advertising. Large scale public display screens present the most interesting of such opportunities. In one such campaign, persons standing in a specified location in Times Square could look up at a building sized screen and see their favorite Disney characters magically appear and start interacting with them in real time. In another example of AR experiential advertising, National Geographic produced one with wild animals, dolphins, dinosaurs, astronauts and experience thunderstorms. By standing in a specific spot, users could pet or interact with anything that came up on screen.
AR will change how properties are marketed. It’s one thing to go online a view photos of rooms, but if a passerby can use their smart device to virtually see through the walls of a house it can give them a whole new spatial perspective. Such access may entice curbside prospects to contact the realtor or owner for a showing. The technology can also help a builder or architect show people how a new home will look when built on a particular site or lot.
Today, state of the art telepresence refers to installations in conference rooms that put local attendees on one side of the room facing a bank of TV monitors and cameras that allow them to view remote attendees with similar setups. The idea is to simulate the intimacy of a face to face conversation without having to travel to do so. Skype can be thought of as a manifestation of this for consumers to have similar interaction. Volumetric 3D Sensing technology has been applied to next generation conference room technology referred to as “Immersive Telepresence” which has already made it out of the lab. Mobile AR will deliver personal immersive telepresence through the technologies of wearable displays, mobile devices with volumetric 3D sensors (to capture one’s own image) and high speed data connectivity.
Many people are baffled by the one dimensional instructions that come with items that require assembly such as toys, furniture, electronics and appliances or routine maintenance such as changing a printer cartridge. This results in support calls, customer dissatisfaction and a low likelihood that another such purchase will be made in the future. Providing AR instructional apps that identify and highlight the parts to use, how they are to fit together and what tools to use to join them has the potential to revolutionize how such products are sold and marketed while simultaneously reducing manufacturers’ costs for customer support and returns.
The most common way for companies to provide live technical support for appliances, electronics, toys, etc. is over the phone where the operator is blind and must rely upon what they are told by the customer to bring the situation to resolution. But AR glasses can be the eyes and ears of the remote technician who can then overlay visual information to the customer using techniques like those used with a football commentator’s telestrator by highlighting what button to push, what screw to remove or which way to rotate the screwdriver.
Like many new technologies, investment has come first to military applications where there are already many applications deployed or in development. AR technology has the potential to significantly change the face of military combat and eliminate the home ground advantage.
Frontline Combat: Land, Air, Sea
Heads-up displays and head mounted displays that allow pilot’s to keep their eyes on their objective during flight have been installed in aircraft for decades. This technology has been proven effective and has evolved tremendously over the years.
Communicating situational awareness information to soldiers while a mission is being carried out would be an invaluable application for AR. Information about the terrain, navigation or enemy combatant locations could be aggregated from satellite, airborne and map resources and sent to soldiers based on each one’s own location, orientation and role in the field to provide a huge advantage over the enemy.
AR research is being done that would allow soldiers to identify the location of fellow combatants in the battle theatre who may or may not be out of the direct line of sight. The idea is to reduce incidents of friendly fire through use of head mounted displays that would hover icons or identifying information above soldiers’ locations. This would be of value to airborne and ground borne troops on foot or in vehicles.
The military has vast fleets of vehicles of diverse makes and models. The use of AR has the potential to allow personnel to perform vehicular repair and maintenance more efficiently and effectively. For routine repairs and maintenance there will be applications that guide them through the tasks step by step through glasses that allow them to do so hands free while listening to verbal instruction. For more complex operations a remote specialist can make use of the view through a camera integrated into the HMD of the on-site technician to see what they are seeing, speak to them through an integrated earpiece and use pointers and mark-ups projected on the view of the on-site technician to guide them through the procedure. This would be useful for emergency field repairs as well when there is no technician present.
Similar to the mechanical scenarios above, AR can be used for Medical Support in the Battlefield in analogous situations. AR can walk non-medical personnel through routine procedures such as setting a bone or applying a tourniquet or be used to allow remote specialists to remotely assist in a surgery. This scenario has been undergoing testing for several years.
Soldiers can take combat training to a more realistic level with AR than virtual reality simulators can provide. Outfitted with AR glasses at training grounds and facilities, people can be made to appear from behind various walls and barriers forcing the trainee to make friend-or-foe and combatant/non-combatant judgments with their weapons. Such an AR application could be adaptive in respect to the users’ performance by getting progressively more challenging or providing additional scenarios that need to be practiced more. The app could pick up where a user left off in the previous exercise and continue to provide a different experience each time used.
Porn has historically been a driving force behind the adoption of many technologies and AR shall not be an exception. It does not take much imagination to speculate what these applications May be.
Print magazines have long been losing subscribers and advertising revenue to the richer media of the Web. Publishers are attempting to stem the tide by using magazine photos as AR markers to trigger 3D video content that brings the models to life on the page surface.
Apps will be developed for the simulation of sex acts for the purpose of private individual arousal.
Apps will be developed for the purpose of overlaying lewd imagery on the bodies of unwitting participants. This will surely trigger outcry over privacy concerns as usage of this technology in inappropriate venues like schools and the workplace. Society will be forced to put in place moral and legal standards around acceptable use as it has with other such technologies.