[Updated 7 january, 2015]
My preceding posts have been about defining augmented reality, how it’s used and how it works. From here I intend to cover the state of the AR industry. This includes how it is evolving, how it is organizing, how standards are arising, how it is collaborating, and who the players are.
I think a good place to start this section would be to cover the ongoing process of standards development. Without standards, the industry would be a fragmented and disorganized mess, doomed to never being able to fulfill the technology’s great promise. However, my research indicates that the stakeholders have been doing an excellent job of carefully and deliberately collaborating to design the arena in which they shall all compete. It’s exciting to be here at the dawn of the era where I can witness the hardware and software technologies converge and the new technology begin to emerge.
Why Standards are Important
“The international language of commerce is standards. Adherence to agreed upon product or service specifications underpins international commerce, enabling trillions of dollars of goods to flow across borders, regardless of the spoken language of any business parties. The common acceptance of standards is fundamental to the success of robust, fair and free trade. Without standards, it would be difficult to imagine the tremendous volume and complexity of international trade.”
Donald L. Evans, Secretary of Commerce, 2004
Voluntary consensus standards are at the foundation of the global economy. Standards systems promote the public good, enhance the competitiveness of industry, and contribute to a liberalized global trading system. This “essential infrastructure” is therefore important to everyone, and it is important that everyone understand that and work towards maintaining and improving the system. A standards system is strengthened whenever standards developers share a common vision for meeting stakeholder needs. It is important for public and private sectors to share this common vision, since they provide the people, the resources, the technical contributions, and the intellectual vitality that underpin the standards system.[StandardsPortal.org]
Standards are needed in the augmented reality industry for several reasons.
- Widely adopted standard interfaces lead to interoperability of hardware and software which is necessary to grow market opportunity for the whole industry.
- Standards obviate the need for each company to reinvent the wheel. This allows solutions to be developed inexpensively and move to market more quickly.
- All players benefit from cooperation in building a foundation for the industry to compete upon. Fragmentation of the technology landscape benefits no one.
- The compelling use cases for augmented reality clearly ride on mobile devices. The mobile device industry relies on standards for nearly every facet of device hardware. AR standards need to abide by and gow with this ecosystem.
The Qualities of Good Standards
Standardization systems should be based on the following set of globally accepted principles:
- Transparency – Essential information regarding standardization activities is accessible to all interested parties.
- Openness – Participation is open to all affected interests.
- Impartiality – No one interest dominates the process or is favored over another.
- Effectiveness and Relevance – Standards are relevant and effectively respond to regulatory and market needs, as well as scientific and technological developments.
- Consensus – Decisions are reached through consensus among those affected.
- Performance Based – Standards are performance based (specifying essential characteristics rather than detailed designs) where possible.
- Coherence – The process encourages coherence to avoid overlapping and conflicting standards.
- Due Process – Standards development accords with due process so that all views are considered and appeals are possible.
- Technical Assistance – Assistance is offered to developing countries in the formulation and application of standards.
The process to develop and implement standards should be:
- Flexible to allow the use of different methodologies to meet the needs of different technology and product sectors
- Timely so that purely administrative matters do not result in a failure to meet market expectations
- Balanced among all affected interests.
High Level Goals for Augmented Reality Standards
- Integration of sensor processing, vision, computation and rendering
- Standards should envelop
- cloud services integration
- application software
- sensor network integration
- web browsers
- 3D asset authoring and delivery
- platform APIs
- sensors hardware integration
- Seamlessly integrate content from multiple sources
- Run in any browser on any device
- Smoothly run on mobile devices at high framerates (i.e. 60 Hz) and consume a minimal amount of power (i.e. 500 mW)
Key Standards Groups for AR
An increasing number of industry groups and standards-development organizations have begun work on open frameworks and extensions of existing standards that have implications for augmented reality. Here is a rundown of those directly influencing the emergence of AR.
The Khronos Group is a not for profit, member-funded consortium focused on the creation of royalty-free open standards for parallel computing, graphics and dynamic media on a wide variety of platforms and devices. It was was founded in January 2000 by a number of leading media-centric technology companies dedicated to creating open standard APIs to enable the authoring and playback of rich media on a wide variety of platforms and devices. All 100+ Khronos members are able to contribute to the development of Khronos API specifications, are empowered to vote at various stages before public deployment, and are able to accelerate the delivery of their cutting-edge 3D platforms and applications through early access to specification drafts and conformance tests. Several of the standards under development have implications for AR such as: Open CL, Open GL and Open GL|ES for visual computing; glTF and Collada for 3D asset handling; StreamInput, OpenKCam and OpenVX for Sensor Processing; WebGL and WebCL for browser interaction. (These standards will be covered more in depth in my next post.)
The world’s largest professional association dedicated to the lofty goal of advancing technological innovation and excellence for the benefit of humanity. IEEE and its members comprise a global community through publications, conferences, technology standards, and professional and educational activities. A division called IEEE Computer Society is concerned with all major areas of computing and information technology. IEEE Computer Society Visualization & Graphics Technical Committee (VGTC) is a subdivision of this that is focused on promoting the immediate and long-range challenges in visualization and computer graphics and related research areas including AR. Membership in VGTC is open to all individuals interested in visualization and computer graphics at a professional level. It is chaired by members of academia, industry and military and meets annually at different international locations.
OGC (Open Geospatial Consortium)
An international industry consortium of 477 companies, government agencies and universities participating in a consensus process to develop publicly available interface standards. OGC standards support interoperable solutions that “geo-enable” the Web, wireless and location-based services and mainstream IT. The standards empower technology developers to make complex spatial information and services accessible and useful with all kinds of applications. OGC is promoting the Augmented Reality Markup Language (ARML) 2.0 to provide an interchange format for Augmented Reality applications to describe and interact with objects in an AR scene, with a focus on mobile, vision-based AR. The OGC ARML 2.0 Standards Working Group has set a specific focus on a generic way to describe Trackers in an Augmented Reality environment, allowing users to define a multitude of Trackers, such as Marker-Trackers, Face-Trackers, Natural Feature Trackers and even non-vision-based Trackers, to allow a multitude of AR use cases to be implemented with ARML 2.0.
W3C (World Wide Web Consortium)
The W3C mission is to lead the World Wide Web to its full potential by developing protocols and guidelines that ensure the long-term growth of the Web. Five leading global organizations jointly signed an agreement to affirm and adhere to a set of Principles in support of The Modern Paradigm for Standards; an open and collectively empowering model that will help radically improve the way people around the world develop new technologies and innovate for humanity. The W3C Augmented Reality Community Group is an open forum for collaborative discussions about the intersection of Augmented Reality and the Web, or more simply the Augmented Web. The Augmented Web embraces the changes brought about by HTML5 and other related standards including Geolocation, DeviceOrientation, DeviceMotion, WebGL, Web Audio, Media Capture & Streams and WebRTC. The Augmented Web integrates all of these disparate technologies into an integrated new vision of the web. This group will not produce specifications. Instead it aims to build an integrated community voice that reaches out to all of the other relevant working groups and standards bodies to ensure that the Augmented Web perspective is clearly represented and considered. Their goal is to help ensure that the disparate standards and APIs being planned and implemented by these other groups can be seamlessly integrated into this new vision for the Augmented Web.
OMA (Open Mobile Alliance)
OMA was formed in June 2002 by the world’s leading mobile operators, device and network suppliers, information technology companies and content and service providers. OMA delivers open specifications for creating interoperable services that work across all geographical boundaries, on any bearer network. OMA’s specifications support the billions of new and existing fixed and mobile terminals across a variety of mobile networks, including traditional cellular operator networks and emerging networks supporting machine-to-machine device communication. OMA is the focal point for the development of mobile service enabler specifications, which support the creation of interoperable end-to-end mobile services. For instance, the OMA Mobile Augmented Reality (MobAR) v1.0 defines an overall framework that enables mobile Augmented Reality services. The core functionalities exposed by the MobAR Enabler include: personalization and management of AR Content, user interactivity handling to AR content, network and client APIs as well as security and privacy.
OpenCV (Open Source Computer Vision Library)
OpenCV is an open source computer vision and machine learning software library built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in commercial products. Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code. The library contains algorithms that can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high resolution image of an entire scene, find similar images from an image database, remove red eyes from images taken using flash, follow eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc. OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 7 million. The library is used extensively in companies, research groups and by governmental bodies. Along with well-established companies like Google, Yahoo, Microsoft, Intel, IBM, Sony, Honda, Toyota that employ the library, there are many startups such as Applied Minds, VideoSurf, and Zeitera, that make extensive use of OpenCV.
Web3D (Web3D Consortium)
The Augmented Reality Working Group focuses on utilizing and extending X3D (Extensible 3D) capabilities to support augmented reality (AR) and mixed reality (MR) applications. They endeavor to collect requirements and describe typical use cases for using X3D in AR/MR applications, produce and propose X3D components for AR/MR scenes and applications and produce sample AR/MR applications using X3D to demonstrate how this functionality can work correctly. X3D is a royalty-free and openly published standard file format specification and run-time architecture to represent and communicate 3D objects, events, behaviors and environments. The X3D suite of International Standards Organization (ISO) ratified standards provide a robust abstraction for the storage, retrieval and playback of real time 4D graphics content across diverse platforms. Web3D is working toward interoperability with other AR standards such as W3C Augmented Reality Community Group, OGC ARML WG, AR Standards Group, ISO-SC24 and KHRONOS.
ARWave is a open source project to create a standard method for geolocating data on Wave servers. The WFP protocol provides an ideal method to aggregate and share data for augmented reality uses and will help lay the foundations for an open, universally accessible, and decentralized system for shared augmented reality overlays which various clients can connect to and use. This AR Network should spark a lot more rapid adoption of AR technologies, give existing browsers more functionality, and provide the network infrastructure, allowing much of the fictional depictions of AR to become a reality one day.
ISO (International Organization for Standardization)
The world’s largest developer of voluntary International Standards. International Standards give state of the art specifications for products, services and good practice, helping to make industry more efficient and effective. JTC 1 is the standards development environment where experts come together to develop worldwide Information and Communication Technology (ICT) standards for business and consumer applications. Additionally, JTC 1 provides the standards approval environment for integrating diverse and complex ICT technologies. SC 24 is the Standards Committee for computer graphics, image processing and environmental data representation. WG 9 is the Working Group for the Augmented RealityContinuum (ARC) concepts and reference model.
MPEG (Moving Picture Experts Group)
A working group of ISO/IEC in charge of the development of international standards for compression, decompression, processing, and coded representation of moving pictures, audio and their combination. MPEG is advancing two standards for augmented reality: The first is the MAR (Mixed and Augmented Reality) reference model which aims at defining the domain of mixed/augmented reality, the main concepts, various terms and their definitions, and an overall system architecture analyzed from various viewpoints, principles on which the document will be built. MAR reference model specifies a set of minimum functions and performance levels that should be supported by an MAR system. The reference model contains a list of possible use cases and their validation with respect to the architecture proposed by the reference model. The second standard is Augmented Reality Application Format (ARAF) which is a collection of a subset of the ISO/IEC 14496-11 (MPEG-4 part 11) Scene Description and Application Engine standard, combined with other relevant MPEG standards (e.g. ISO/IEC 23005 – MPEG-V), designed to enable the consumption real-time 2D/3D hybrid content. The Second edition particularly focuses on tools such as 3D video, 3D audio, image and video descriptors for recognition and tracking and server side processing.
MIPI Alliance (Mobile Industry Processor Interface)
The mission of the MIPI Alliance is to benefit the entire mobile industry by establishing standards for hardware and software interfaces in mobile devices. The organization actively promotes and encourages the adoption of these specifications throughout the industry value chain. Adopting a set of standard hardware interfaces will produce peripheral hardware products from multiple vendors which work seamlessly with numerous processor and system-on-a-chip products. Since software is an integral component of many of these interfaces, some level of software standardization will enhance this improved inter-connectivity. A common set of interfaces will maximize design reuse, drive innovation and reduce time-to-market for all participants. MIPI Alliance’s vision is to develop the world’s most comprehensive set of interface specifications for mobile and mobile-influenced products. The role of the MIPI Alliance is complementary to other industry organizations. While MIPI are not themselves advancing any AR standards, they do advocate for standards adoption for the benefit of the industry.
JEDEC (Joint Electron Device Engineering Council)
The JEDEC Solid State Technology association works to develop open standards for the global microelectronics industry, with more than 3,000 volunteers representing nearly 300 member companies. They bring manufacturers and suppliers together to participate in more than 50 committees and subcommittees, with the mission to create standards to meet the diverse technical and developmental needs of the industry. These efforts ensure product interoperability, benefiting the industry and ultimately consumers by decreasing time-to-market and reducing product development costs.
AR Community for Open and Interoperable Augmented Reality Experiences is hosted by Perey Research and Consulting for the purpose of advancing open and interoperable augmented reality. This meta-community seeks to monitor progress and activities across a wide variety of standards development organizations (SDO) and industry groups whose technologies have a relationship to AR. Their goal is to advocate a neutral platform on which leaders of various initiatives can explore ways to coordinate their activities, and to reconcile complementary and conflicting work by these parties. They also provide guidance to SDOs and communities interested in open and interoperable AR. AR Community has posted a page where most of the standards being developed by the other organizations mentioned in this post are listed which can be found here.
In addition to these standard setting organizations, there are a number of other organizations that are tangentially concerned with the advancement of augmented reality such as: SIGCHI, SID, NISO, ETSI, 3GPP, BSI