- Open Access
IoT + AR: pervasive and augmented environments for “Digi-log” shopping experience
© The Author(s) 2019
- Received: 21 July 2018
- Accepted: 22 December 2018
- Published: 4 January 2019
The current bare Internet of Things (IoT) infrastructure has recently been extended to include smarter and more effective user interactions. Individual or meaningful sets and groups of IoT objects can be imbued with data and/or content in a distributed manner and efficiently utilized by the client. This distribution makes it possible to scale and customize interaction techniques such as augmented reality (AR). This paper proposes an architecture for combining the AR interface with IoT for an improved shopping experience. The proposed architecture is distributed and therefore scalable such that any IoT product can be accessed on the spot locally without any server restriction and provide intuitive AR-based visualization and interaction for a flexible product trial in the showroom. We identify three key architectural components required to support such a seamless and scalable AR service and experience for IoT-ready products: (1) object-centric data management and visualization, (2) mechanism for accessing, controlling, and interacting with the object, and (3) content exchange interoperability. We illustrate the possible scenarios of shopping in the future with the interactive and smart digital information combined with the analog, that is, the real world. A proof-of-concept implementation is presented as applied to such a “digital–analog” style of shopping. In addition, its usability is experimentally assessed as compared to using the conventional control interface. Our experimental study shows that the subjects clearly experience higher usability and greater satisfaction with AR-interactive shopping, thereby demonstrating the potential of the proposed approach.
- Augmented reality (AR)
- Internet of Things (IoT)
- User experience
Augmented reality (AR) and the Internet of Things (IoT) have received significant attention as the key technologies for making our future living spaces smarter, more responsive, and more interactive, thereby changing our everyday lives [1, 2]. AR is a type of interactive medium that provides a view of the real world augmented by, and/or spatially registered with, useful computer-generated information. It empowers users to understand the world and amplify their intelligence in solving problems and conducting various tasks [3, 4]. In other words, AR offers a convenient approach for users to visualize and interact with physical objects and their associated data. In addition, a spatially registered and visually augmented interface offers a direct and semi-tangible interface, and is thus easy to comprehend and highly useful, particularly for everyday and/or anywhere usage . For example, Microsoft showcased a future AR service using a pair of mixed reality smart glasses, which directly visualize and make the object’s functionality interact with datasets from physical objects and structures within the user’s environment .
In addition, recent IoT, as an infrastructure for “everywhere” services, offers an efficient way of managing the necessary and massive amounts of associated data (for example, individual product information) in a distributed and object-centric fashion . IoT refers to a network of everyday physical objects embedded with minimal computing elements for the sensing, collecting, and/or sharing of data, and even controlling the objects themselves, such as electronic products. Such an infrastructure has been touted as the basis for the future smart environments through an intuitive control and context-based services .
These two seemingly unrelated concepts, AR and IoT, might have different objectives, but can be complementary to each other along with the potential advantages and expected synergies of integrating them . AR provides an intuitive method for users to visualize and interact with IoT objects and their associated data. In particular, context-aware AR services are made possible by using and tapping into the more refined environment information made available by the IoT infrastructure . In fact, it can also provide a natural environment for combining the convenience of interactive digital information (e.g., AR-enabled) to a more effective, humane, and tangible/physical analog objects/world. In the midst of everything going digital, analog is making a comeback in our daily lives with the recent popularity of printed books, vinyl records, and film-based photos.
In the previous studies conducted in the IoT or AR field, many people suggested everywhere data management and intuitive visualization such as a server-based approach to ubiquitous AR services with everyday physical objects. However, because the object recognition process, equivalent to looking up the content directory, involves complicated feature matching, for a vast number of objects, expanding the AR services to large everyday spaces has been difficult to achieve. In the cloud services for providing an AR service, it was difficult to provide scalability to the IoT object . Thus, many researchers have focused on AR applications to carry and share the useful information connected with physical objects, and an enhanced AR system allows a user to connect to objects.
In line with such thinking, i.e., the idea of synergistic marriage of AR and IoT, this paper presents a new AR shopping framework and experience enabled by an extension of the IoT as a control and product trial interface; in addition, we demonstrate our proposal using an actual prototype system and validate our claims in terms of its improved usability and system performance. We illustrate possible scenarios of shopping in the future with the interactive and smart digital information combined with the analog real world and present that as a proof-of-concept enables to immediately obtain information about shopping items and correctly visualize the information based on the exact location of the item to an AR client. The proof-of-concept implementation is presented as applied to such a “digital–analog” style of shopping. In addition, its usability is assessed experimentally as compared to using the conventional control interface.
Additionally, object tracking is a fundamental problem in AR. The proposed IoT products can also be to easily apply recognition and tracking for AR. Besides generic data and service content, individual IoT products of interest in the vicinity of the AR client can communicate the information required to recognize and track itself, including the features, algorithm type, and even the physical condition (for example, lighting, distance, or other companion reference object). That is, the AR client is “guided” by the target object itself to localize and track it . Note that, in this scheme, the number of candidates in the matching, i.e., only the candidates in the interaction space of the AR client or user, is relatively low. This, in turn, makes it feasible to use a collective algorithmic method and reduce the number of features, templates, and models in the matching process, further lowering the data requirement.
The rest of this paper is organized as follows. First, we provide a review of related research and requirements of our proposed IoT + AR architecture in Section II. In Sections III and IV, we discuss futuristic use-case scenarios and a detailed data flow of IoT + AR for shopping situations. Section V presents the actual implementation and Section VI presents the validation usability experiment. Finally, in Section VII, we summarize our study and conclude the paper with a discussion and directions for future work.
The review of related research focuses on three key components and requirements in our proposed IoT + AR architecture, namely the current state-of-the-art on AR/IoT data management, previous approaches to interaction with IoT objects including the few cases of using AR, and standard content representation or system interoperability protocols.
AR data and content representation for physical objects
AR services commonly need to manage generic data and service contents for their constituent objects or augmentation targets, which are physical everyday objects. Herein, we review the current approaches for representing such physical object data for AR use (for example, the architecture and data-handling).
Previous AR systems were implemented as a single application with all of the content and assets embedded in it, using programming libraries [11, 12]. As such, the augmented content of an object tended to be simple (for example, to simply demonstrate the augmentation capability) and unorganized. GPS-equipped mobile and smartphones have allowed for location-based geographical and AR services to be developed, for example, providing guidelines for commercial points of interest and tourism [13, 14].
Such a service has necessitated the separation of content (and its format specifications) and the underlying player to support the notion of “everywhere” content and service, as well as a unified management of content on the server. HTML , KML , and ARML  are markup languages for such purpose. For example, Wikitude proposed the augmented reality markup language (ARML) for location-based services . ARML allows defining geographical points or landmarks of interest and associating GPS coordinates and simple augmentation content (for example, text, logos, and images). Several other content representation methods exist for AR services, but they require either a specific application or content type (for example, video-based , AR on-line manual , and AR guide ) or complicated scripting without sufficient abstraction. However, a standard interoperable content format for representing various comprehensive forms of AR services is yet to be proposed.
In our case, the client AR system receives “feature datasets” and “contents” information for each shopping item from the discovered IoT device in the user proximity in the standard format (e.g., front images of shopping items, product origin, and price) to recognize, visualize, and interact with the item . In addition, information exchanged between the AR client and the shopping item is contained in the IoT device rather than retrieved from an external server. We assume that the future IoT object will have this information (feature datasets for AR tracking, generic contents, UI control structure) as a standard format. We can envision that different IoT objects may contain different AR information depending on its characteristics, for example, functionalities, process interfaces, and operating manuals.
AR data/content storage, management, and indexing for physical objects
The most frequently used method of viewing and interacting with digital objects and products is to use interfaces as provided by hand-held remote controls; more recently, the smartphone and GUI-based interfaces have replaced it quickly . AR provides a tighter augmentation through the process of target object recognition and identification . Although a server-based approach to ubiquitous AR services with everyday physical objects is possible, the AR services for large everyday spaces has been difficult to achieve owing to the object recognition process, complicated feature matching, and content look-up for several objects.
High-performance cloud computing services exist for the fast object matching process and expediting the associated content retrieval in providing an AR service . Nevertheless, it will still be difficult to support the scalability to the level of “everywhere.” An alternative may be to connect to a singular areal server (serving only a particular local area such as a single home) managing only a limited number of objects [26, 27]. This is similar to the concept of fog computing to enable computing services at the edge of the adjacent network for effective data management. For example, Rathore and Park presented a fog-based attack detection framework to detect attacks in IoT. This approach was suggested to solve the problem that cannot produce significant results at the centralized attack detection mechanisms due to scalability, distribution, resource limitations, and low latency . Sharma et al. proposed a fog node architecture to mitigate security attacks for real-time analytic services .
Thus, a filtering approach (to reduce the search space), such as broadcasting messages, to nearby clients was proposed . Iglesias et al. suggested a method for identifying and augmenting candidate target objects with contextual data such as the user’s attribute, user-object proximity, relative orientation, resource visibility, and making the final selection manually . Ajanki et al. proposed a similar concept . Unfortunately, there has been no noticeable work on scaling AR services and their efficient data management to large-scale everyday environments (for example, an AR service that operates at home, the workplace, on the street, or in a shopping area). The approaches (including even the future “unified” Web-based solution) outlined above are based on the central network server architecture, as already indicated, and incur a serious performance bottleneck. Therefore, a few studies are attempting to obtain datasets directly from objects close to the user in the same space .
As already mentioned, in our scheme, the AR-enabling IoT device itself already stores and contains (in-house) standard “features” information used for the mobile AR clients to recognize/track them and can communicate generic contents for various shopping purposes including the augmented display. Thus, an interactive control of the IoT device is possible on the spot. To select among millions of different objects with a filtering approach, the AR client finds IoT objects (equipped with elementary processing, storage, and network modules) similar to identifying mobile access points, which communicate the necessary AR tracking information to the client. Because there is bound to be only a relatively few target objects around, the AR client can quickly identify (and even track) the objects and retrieve the associated content.
AR interaction for physical objects
The current and most prevalent application of AR offers an excellent control method for in situ object control (or even for remote objects using a remote-controlled camera) . AR can be used to visualize simulations of applied control for previewing or even training purposes [32, 33]. However, there have been only a few attempts of using AR (or even VR) as the control and simulation interface.
As the first proposed result, Rekimoto and Ayatsuka proposed a visual tagging system, called a CyberCode , which uses 2D barcodes to identify and detect objects and offers different methods to manipulate physical objects. For example, the user can metaphorically “drag-and-drop” one object onto another to invoke a certain functionality (for example, by dragging and dropping a projector object onto a computer, the computer will retrieve the currently projected slide). Similarly, Heun et al. proposed an AR interface to create new functionalities of smarter objects that have an embedded processor and communication capability .
In addition, the Microsoft HoloLens platform suggested and presented a situation that visualizes datasets associated with objects (e.g., motor temperature and door functioning) to reduce the maintenance costs of a particular product (e.g., elevators) . However, datasets in the cloud need to continuously manage the updated information, and when there are many similar objects, it becomes confusing what it is. On the other hand, using AR that contains datasets in each object, it would be more intuitive to visualize information directly at the precise position related to the object.
In addition, Muller et al. proposed an interactive AR-enabled appliance instruction manual . In their prototype, an AR-capable device can interact with an appliance through a pre-established connection. Lifton and Paradiso presented a dual reality system, realizing an interplay between the physical and corresponding mirrored and simulated virtual worlds ; here, interactions in the real world were reflected onto the virtual world. Lu proposed a bi-directional mapping technique for enhanced information visualization. For example, when a user turns on an appliance in a real environment (for example, a TV), the attribute of the deployed sensors detects the user’s activity and transmits it to the simulated world. The virtual world can also generate counterpart representations of the real world. This system was developed to realize eco-feedback for energy-saving . More recently, Alce et al. proposed a comparison of three basic AR interaction models (floating icons, floating menu, and WIM) for managing IoT environments, and found that the WIM model stood out as difficult and time-consuming [36, 37]. In our case, we evaluated AR interaction methods using a mobile-type smartphone that connects to the IoT products with pre-configured information about itself in its memory.
Despite the potential of such AR interfaces (e.g., over the conventional remote-controlled types), it is not clear how a consistent and coherent AR interaction framework should be established for “millions” of different objects. In the previous studies mentioned above, AR interfaces are mostly anecdotal and designed in an ad-hoc manner.
We illustrate two use-case scenarios using AR-capable shopping objects in terms of emphasizing the effectiveness of our AR-enabling IoT approach, which highlight the three aforementioned key components: (1) object-centric data management and visualization, (2) mechanism for accessing, controlling, and interacting with the object, and (3) content exchange interoperability.
Test driving at the showroom
Besides sensing, collecting, and exhibiting useful data, IoT objects are meant to be digitally “controlled” to realize related smart services [35, 36]. In many situations (and in scenario 1 as well), direct in situ control is needed, and AR is a proper interface (for example, versus a simple GUI-based control button interface) because it provides the necessary contextual information to make the task easier and the situation more clearly understood . For instance, IoT devices with connectivity and computing capability can be embedded in an object as sensor systems. Thus, objects themselves can communicate the necessary data such as current sensor information of a physical device to the client on a need-to-know basis (including the information required for recognition and tracking). That is, the data are now delegated and distributed to individual objects in the environment.
Therefore, our approach provides an ideal and natural infrastructure for “everywhere” AR accessibility with physical shopping objects. Note that the data and/or content can also be uploaded to the objects for adding and creating new shopping services and applications. Besides generic data and service content, individual objects of interest in the vicinity of the AR client can communicate information required to recognize, visualize, and interact itself, including the features and shopping contexts.
Step-by-step how-to-use guideline of a product
This scenario illustrates how AR services can be accessed at any time to “everywhere” object and operates in the simulation mode. The client can connect to any object using the assumed standard protocols without the local or remote central server communication. The AR client detects the presence of objects (equipped with elementary processing and operation functionalities) in its vicinity (similar to identifying Wi-Fi access points) [37, 38]. These objects communicate the necessary information to the client with intuitive AR visualization to provide direct overlapping situations, and objects having their own operation can be distributed, stored, and exchanged to the AR client.
Because there is bound to be only relatively target objects around, the client can quickly identify (and even track) the objects and retrieve the associated visualization content. It can be argued that the objects simply need to be organized geographically and managed through a hierarchical network of servers (similar to the case of a geographical service). However, disregarding the enormous number of objects to be handled (even compared to that of geographical objects), there is currently no common technology for accurately recognizing individual objects (which may be mobile) in indoor locations without pre-registration of their tracking features.
So far, we have described the motivation, futuristic scenarios, and technical aspect of realizing the IoT + AR platform as applied to the offline interactive shopping situation. The underlying assumption is that our proposed approach is useful and well-received and creates an effective shopping experience. Thus, in this section, we experimentally assess its satisfaction level and usability.
The first experiment analyzed the level of user satisfaction by showing two types of interfaces on a hand-held device: (a) conventional web-based and (b) AR-based. The level of the user’s overall satisfaction (10 participants, average age of 36) was evaluated through a survey question on a 7-point Likert scale. The Wilcoxon test for paired samples revealed that the mean satisfaction score was significantly higher (Z = 2.871, P < 0.05) with the AR (average 6.2) than the conventional interface (average 3.5).
Subjective usability survey assessing the ease of use, naturalness, fatigue, speed, and simple preference
Answered after experiencing two conditions [(a) conventional switchable interface and GUI-based button interface and (b) IoT + AR interface]
How accessible did you find the interface to be?
(1: very difficult – 7: very easy)
How intuitive and natural did you find the interface to be?
(1: very contrived – 7: very natural)
How fatigued were you after using the interface?
(1: very fatigued – 7: not fatigued at all)
How fast did you feel you were able to complete the task?
(1: very slowly – 7: very fast)
Which interface do you prefer?
In this paper, we described how the current AR infrastructure can be extended to include smarter and more effective user interactions for physical objects in real analog shopping situations. Individual or groups of physical IoT objects can be imbued with data and/or content in a distributed manner and efficiently utilized by the AR client along with the potential advantages and expected synergies of integrating them. The distribution makes it possible to scale and customize interaction techniques such as AR. Our approach leverages on the IoT control interface for physical objects, and intuitive and natural AR interaction in a complementary way, also combining the digital and analog worlds. Thus, our notable approaches to their integration into the IoT framework as a control interface can enable the given AR service to significantly reduce latency. Through the pilot experiments, we also partly validated our claims of the synergy and advantages of our proposal. An outstanding issue is that the contents and data exchange protocol need to be standardized for true scalability. In the future, we will continue to further demonstrate our approach to a large-scale shopping center and investigate how to effectively put AR information in such a large space related to a real application in a shopping experience to provide an impression of our work. Especially, we plan to validate our approach in terms of AR object recognition for improving the shopping experience. In addition, we will develop a particular AR interface that can be adaptively tailored to such objects given the client platform.
First author is DJ, contributing to Use Case Scenarios, prototype implementation, and usability experiments sections. Corresponding author is GJK, contributing to all sections of the manuscript. Both authors read and approved the final manuscript.
This paper was supported by Wonkwang University 2018.
The authors declare that they have no competing interests.
Availability of data and materials
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Jo D, Kim GJ (2015) Local context based recognition + Internet of Things: complementary infrastructures for future generic mixed reality space. In: Proceedings of the 21st ACM symposium on virtual reality software and technology (VRST 2015). pp 196Google Scholar
- Gimenez R, Pous M (2010) Augmented reality as an enabling factor for the Internet of Things. In: W3C Workshop: augmented reality on the webGoogle Scholar
- Brooks FP (1996) The computer scientist as toolsmith. Commun ACM 39(3):61–68View ArticleGoogle Scholar
- Radkowski R (2013) Augmented reality to supplement work instructions. Model-Based Enterprise Summit, 2013Google Scholar
- Kasahara S, Heun V, Lee A, Ishii H (2012) Second surface: Multi-user spatial collaboration system based on augmented reality. SIGGRAPH Asia 2012 emerging technologies, 2012. pp 1–4Google Scholar
- HoloLens (2017) MS HoloLens. https://www.microsoft.com/en-us/hololens. Accessed 19 Nov 2018
- Jo D, Kim GJ (2016) ARIoT: scalable augmented reality framework for interacting with Internet of Things appliances everywhere. IEEE Trans Consum Electron 62(3):334–340View ArticleGoogle Scholar
- Perera C et al (2014) Sensing as a service model for smart cities supported by Internet of Things. Trans Emerg Telecomm Technol 25(1):81–93View ArticleGoogle Scholar
- Jeong Y, Joo H, Hong G, Shin D, Lee S (2015) AVIoT: web-based interactive authoring and visualization of indoor Internet of Things. IEEE Trans Consum Electron 61(3):295–301View ArticleGoogle Scholar
- Rambach J, Pagani A, Stricker D (2017) Augmented things: enhancing AR applications leveraging the Internet of Things and universal 3D object tracking. IEEE international symposium on mixed and augmented reality 2017Google Scholar
- Vuforia (2017) https://developer.vuforia.com. Accessed 19 Nov 2018
- ARToolKit (2003) https://en.wikipedia.org/wiki/ARToolKit. Accessed 19 Nov 2018
- Layar (2009) https://www.layar.com/. Accessed 19 Nov 2018
- Wikitude (2008) http://www.wikitude.com/. Accessed 19 Nov 2018
- W3C (2017) World wide web consortium. http://www.w3.org/. Accessed 19 Nov 2018
- KML (2017) Keyhole Markup Language. https://developers.google.com/kml/. Accessed 19 Nov 2018
- ARML (2017) Augmented Reality Markup Language 2.0. http://www.opengeospatial.org/standards/arml. Accessed 19 Nov 2018
- TELLME (2015) http://www.tellme-ip.eu/#home. Accessed 19 Nov 2018
- Muller AL, Krußen L (2013) GuideMe: A mobile augmented reality system to display user manuals for home appliances. Advances in computer entertainment (ACE)Google Scholar
- ARLEM (2015) Augmented reality learning experience models. http://arlem.cct.brookes.ac.uk/. Accessed 19 Nov 2018
- Shah S, Yaqoob I (2016) A survey: Internet of Things (IOT) technologies, applications and challenges. smart energy grid engineering (SEGE). pp 381–385Google Scholar
- Physical web (2014) Google’s Physical Web. https://google.github.io/physical-web/. Accessed 19 Nov 2018
- Ahn SC, Go HD, Yoo BH (2014) Webizing mobile augmented reality content. New Rev Hypermedia Multimedia 20(1):79–100View ArticleGoogle Scholar
- Heun V, Hobin J, Maes P (2013) Reality editor: Programming smarter objects. Ubicomp. pp 307–310Google Scholar
- Huang Z, Weikai L, Pan H, Christoph P (2014) CloudRidAR: A cloud-based architecture for mobile augmented reality. Workshop on Mobile augmented reality and robotic technology-based systems. pp 29–34Google Scholar
- Lu CH (2015) IoT-enhanced and bidirectionally interactive information visualization for context-aware home energy savings. Mixed and augmented reality—media, art, social science, humanities and design (ISMAR). pp 15–20Google Scholar
- Kim S, Choi B, Jeong Y, Hong J, Kim K (2014) Novel hybrid content synchronization scheme for augmented broadcasting services. ETRI J 36(5):791–798View ArticleGoogle Scholar
- Rathore S, Park JH (2018) Semi-supervised learning based distributed attack detection framework for IoT. Appl Soft Comput 72:79–89View ArticleGoogle Scholar
- Sharma PK, Rathore S, Jeong Y-S, Park JH (2018) Energy-efficient distributed network architecture for edge computing. IEEE Communications magazine. pp 2–9Google Scholar
- Iglesias J (2012) An attribute-based reasoning strategy to enhance interaction with augmented objects. In: Proc. of esIoT. pp 829–834Google Scholar
- Ajanki A, Billinghurst M, Gamper H, Kandemir T, Kaski S, Koskela M, Kurimo M, Lasksonen J, Puolamaki K, Ruokolainen T, Tossavainen T (2011) An augmented reality interface to contextual information. J Virt Real 15(2–3):161–173View ArticleGoogle Scholar
- Zhu Z, Branzoi V, Wolverton M (2014) AR-Mentor: Augmented reality based mentoring system. In: Proceedings of the 13rd IEEE international symposium on mixed and augmented reality (ISMAR). pp 17–22Google Scholar
- Raskar R et al (2004) RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors. Trans. Graphics 23(3):406–415View ArticleGoogle Scholar
- Rekimoto J, Ayatsuka Y (2000) CyberCode: designing augmented reality environments with visual tags. In: Proc. of DARE. pp 1–10Google Scholar
- Lifton J, Paradiso JA (2009) Dual reality-merging the real and virtual. Facets of virtual environments (FAVE 09). pp 27–29Google Scholar
- Alce G, Roszko M, Edlund H, Olsson S, Svedberg J, Wallergard M (2017) AR as a user interface for the Internet of Things-comparing three interaction models. IEEE International symposium on mixed and augemented reality (ISMAR), 2017. pp 35Google Scholar
- Scavo G (2016) Augmented reality: the human interface with the industrial Internet of Things. http://thearea.org/augmented-reality-the-human-interface-with-the-industrial-internet-of-things/. Accessed 19 Nov 2018
- Jung H et al (2015) IDNet: beyond All-IP network. ETRI J 37(5):833–844MathSciNetView ArticleGoogle Scholar
- Raspberrypi (2017) https://www.raspberrypi.org/. Accessed 19 Nov 2018
- Hart SG, Staveland LE, Lowell E (1998) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Advances in psychology. pp 139–183Google Scholar