Integrating body language movements in augmented reality learning environment

Abstract

Due to excessive use of computer and other low activity learning systems in many schools the physical health status of young students is deteriorating at an alarming rate throughout the world and more seriously in Taiwan upon recent national survey. We address this growing global problem with student learning process in mind in this study we examine making a better use of information technology in virtual really under our new augmented reality (AR) enhanced learning method. Under our AR design/approach we make use of body language by integrating learners' interactions with the computer in their learning based set of physical activities in a way that the effectiveness of the method/technique increases their academic performance through a preference selectivity system. Our results collected from in-school extensive experiments and analysis of the results using recently developed seven multimedia learning subscale factors show that those students who begin with more challenging preferences show more interest and also they gain significantly more in their academic achievements than others.

I. Introduction

Using information communication and technology (ICT) to assist learning and facilitate distance learning has been known for decades and shown that be able to create new learning opportunities. The virtual reality (VR), a specialized advanced area of the technology, however has been also expanding significantly enabling the learners to interact with both worlds of virtual and real [1] whilst bringing in considerable improvements into the learning process [2] and other activities (e.g., [3]). The potential enhancements, however is due dividing the virtual concept into three more distinct technology-specific groups of basic VR, mixed reality (MR) and augmented reality (AR) [4, 5].

Historically, the conceptual discovery of AR goes back to the 1960s but its first practical system has been developed only in early 1990s by Boeing Company. Considering that AR involves hardware and therefore more technology intensive than other areas of VR it is natural to see different views, different understandings and therefore contradictory views. One view comes under augmented virtuality (AV), as another name for VR which composes (real) objects all compiled into virtual environments (VE) where their surrounding environment is also virtual. Another view of AR is n association with the Head-Mounted Displays (HMD) equipment. This however limits AR's capability to the sense of sight. To this effect, Azuma's discusses that we should not restrict the AR by its definition to display specific technologies [6]. He, then extends his view so that AR uses other senses such as hearing, touch, sound, and smell [7]. In relation to this Kaufmann [8] points out that VR technology should immerse the user's activities inside a complete virtual environment.

We, however, believe that AR should allow users to be able to see the real world as well as the virtual as it combines all components in form of virtual objects in the system. This should enhance the AR functions with the real-time activities are replaced to a form that users can realize that the virtual and real objects coexist at the same time. We therefore adopt a more generic form of understanding of the AR being surrounded by real environment supporting the most common definition of AR coming from Milgram and Kishino [9] as, 'there is a continuum of real-to-virtual environment in which AR is a specific area within the generic area of MR'.

For using ICT in some earlier application systems we have made use of keyboard-mouse computer assisted instruction (KMCAI) method. This imposes many practical obstacles and therefore great limitations to the learning process whilst the true AR needs to enjoy whole world of flexibility for various interactions with the system that the learners need to apply using their body language.

Considering that the AR technology has been developing continuously for a wide range of new applications with their uses in learning, adolescents and psychological treatments it is of extreme importance to both health and effectiveness of education to our future generations. With respect to the use of AR in education, we feel that AR has the potential to engage the learners and motivate them to explore new interactions between the teaching materials from real world with virtual objects of the AR learning environment. This rather old but ever evolving system proactively enables learners to make use of extensive interactions with the system i.e. the virtual objects and with the virtual learning image in the virtual world, the learner and the real objects in the real world at the system and user level [6, 7]. In terms of evaluation of AR in education, we use Sumadio and Rambli [8] work indicating the participants show a very good feedback and enthusiasm with the AR. A similar work for an AR supported learning system in the high school indicates that using an AR system not only motivates the students to learn faster and better but it also helps to improve their spoken English [10].

In many cases psychological treatment is categorized learning. For psychological healthcare, Gorini et al [11] analyse the effects of VR in the treatment of generalized anxiety disorders. Another recent research reports an exciting result by the applications of VR for patients with Schizophrenia [12]. For the physical health, Schaik et al argue that participants' strongly prefer Virtual Augmented physical activities over traditional physical exercise [13]. Lamounier et al [14] propose to investigate Augmented Reality techniques to provide new strategies to visualize and interpret cardiologic signs. These works facilitate our better understanding of the information generated by students and professionals from the health area and by the patients.

As far as AR in learning is concerned, Chen et al [15] argue that due to their direct involvement in the process students, as a result of higher level of interest, get motivated and then the speed of their learning enhances significantly. These ideal goals, however, can only be achieved under ideal conditions which are normally not available under practical school's limited budgets. This argument is supported by literature and we see variety of sources supporting complexity of the work. To help with understanding this in our earlier work, Hsiao [3], we have demonstrated that students using three types of AR-based physical activities under an AR learning system can help students with significantly higher academic achievements and also they gain a higher degree of positive attitude towards learning considerably higher than those using KMCAI.

In addition, with AR-based learning the students naturally stay healthier due to being active throughout the time and doing physical exercises during the lessons. We then extend this, by putting the learners at the centre of activities if we bring in a new factor of user approach into the process by including some kind of 'user preferences'. Here, we consider system's performance upon user preferences in conjunction with the AR learning tools which help us to evaluate assess if the AR learning environment been easier to operate and easy to see if students feel that the technical aspects of AR learning system is challenging for them or not. Furthermore, we can activate other essential activities of the AR learning system and examine them in conjunction with our approach in system design. Nonetheless, the use in evaluation of AR learning environment and AR user preferences is still limited [8] and requires extensive improvement.

In order to integrate body language movements in the system we classify these movements and associated body gestures into three types of AR physical activities, namely: 'aerobic fitness', 'muscle strength' and 'flexibility fitness' by using predefined specific meaningful postures, duration of the exercise upon the degree of strength doing the exercise.

The rest of this paper is organized as followed. We discuss paper's main idea and our AR learning system in Section 2. Then, in Section 3 we describe the details of setting up a scenario to conduct our basic experiment for a model at the university and trial runs at the school. This is followed by detailed discussions on our extensive results in Section 4 before our conclusions in Section 5.

II. AR learning system

A new approach to the implementation of AR in the educational environment is taken by creating an AR learning system, using the current teaching curriculum, together with physical activity in this study. This system combines learning with three types of physical activity: aerobic fitness, muscle strength and flexibility fitness. One of the reasons to use this AR learning system is to enhance students' learning by practicing a test in the form of a game competition. Another reason is that if we increase students' physical exercises and if students are short of time for physical exercise due to school's limited time schedule [16].

A. Technical settings

While using our AR learning system in the classrooms, students need not to wear a head-mounted display or any other expensive equipment since more high school classrooms in Taiwan are equipped with at least one computer and a projector with a screen. Thus, for this experiment we only need to use a common webcam as extra equipment for using the AR learning system. The webcam is placed in front of the students in order to capture students' gestures and body movements to interact with the AR system.

To activate the sensing area in the system, students are required to wear the 'red glove' as the marker. Further, they also have to do 'jumping', 'stretching' and 'boxing' in order to hit the correct answers up to some certain number of times instead of only hitting once. The webcam will capture a student's gesture and body movement to interact with the system. For example, in the jumping game, students have to jump high enough in order to let their images to touch the right answer in the virtual world (Figure 1). In order to reach some certain exercise levels, the number of times to hit the answer and jumping height could be designed by teachers.

Figure 1

Snapshot of our AR learning system.

B. The system

Figure 2. shows that the AR learning system starts from 'Flash Animation (Test Start)' which is used to attract students' attention by some new technology novelty of audio and visual effects. After the animation, the users have to choose some certain 'Unit' from the subject content and also choose some certain 'Physical Activity' at the stage of A. If students hit the correct answer by using their virtual images to touch the sense area in the virtual world up to some certain number of times by jumping or boxing, then the system will play 'Flash Animation (Correct Answer)'. In the both animations of Correct Answer and Wrong Answer, the information including the correct answer and the feedback of learning will be provided among them.

Figure 2

AR learning system flowchart.

The user in this learning system is expected to answer each question within one minute. Further, for each question there are two players from different groups to compete with each other. After the players finish the first question, another player in the same group will continue another competition on the second question. When the number of the correct answers is over 5, then the system will display 'Flash Animation (Successful Player)'. If all players in the same group finish the competition, then another two groups will replace the former two groups to start the competition. When the number of 'Successful Groups' are over 3, then the system will display 'Flash Animation (Test Pass)'. Normally, there are approximately 30 students in a class so that students are divided into 6 groups and there are 5 students in a group. In a round, two groups will compete with each other so that there are 3 successful groups at the end.

III. Methodology

A traditional Chemistry high school teacher in Taiwan would be expected to teach students the fundamental concepts of Chemistry knowledge from the textbooks at first. Following the first part of teaching activity, then the teacher would provide 'practice' exercises to the students. The 'practice' normally contains interactive communication and discussion between a teacher and the students. In this study, the AR learning system is used for students as an assisted learning tool in the second part of teaching activity, 'practice'. However, the framework of this system algorithm is not specifically designed for the Chemistry and Science curriculum. It is applicable to any subjects and learning systems.

A. Subjects

The results in this study collected from in-school extensive experiments with 419 students from 5 high schools located in the North part of Taiwan. Students all get involved in the AR learning system but they use different type of AR physical activities. Thus, based on the type of AR physical activity, they are divided into three groups: Group AR-Jump used aerobic fitness, Group AR-Stretch used flexibility fitness and Group AR-Box used muscle strength.

B. Experimental tools

In this study, observations, questionnaires, paper-&-pencil tests and interviews are used for experimental evaluation. In order to explore students' preferences towards the use of the AR learning system, a questionnaire is developed for the users' preferences to classify the preferences of learners engaged in the augmented reality learning environment. Furthermore, in order to examine students' progress on academic achievement within the use of AR learning system, pre-test and post-test paper-&-pencil examinations are applied in this study. Both the questionnaires and pre-test and post-test paper-&-pencil examinations are commented on and the items for content validity by four subject teachers from four high schools and one director of Teaching Affairs.

A questionnaire is designed for the user preference in the AR learning system to classify students' preferences toward the augmented reality learning environment. The questionnaire is integrated and modified from the Constructivist Multimedia Learning Environment Survey conducted by Maor [17] and the Preferences for Internet Learning Environment Survey developed by Tsai [18] but modified based on the features of the AR learning environment and finally classified the questionnaire into seven subscales: 'Ease of Use (EU)', 'Challenge (CH)', 'Rewards (RW)', 'Situated Learning (SL)', 'Collaboration (CO)', 'Competition (CP)' and 'Movement (MO)'. The questionnaire consists of seven subscales which are used to assess students' preferences with the use of the AR learning environment. 'Ease of Use' is applied to assess if students feel the AR learning environment is easy for them to use in the technical aspect. 'Challenge' is used to assess if students feel the content in the AR learning system is challenging enough for them. 'Rewards', 'Collaboration', and 'Competition' are to explore the effects of the learning activity within the use of the AR learning system on to students' learning process. Finally, 'Situated Learning' and 'Movement' are used to assess the impacts of the 'Situated Learning' and 'Movement' design in the AR learning system on students' learning. The items in all seven subscales of the preference questionnaire are scored on a five-point Likert scale ("strongly disagree," "disagree," "normal," "agree," "strongly agree"). While for the "strongly agree" response is assigned a score of 5, for the "strongly disagree" response is assigned a score of 1. Therefore, when students obtained higher scores in the scales, it represented that they showed stronger preference towards this feature. The reliability coefficients (Cronbach's alpha) for these seven subscales are 0.929, 0.927, 0.919, 0.943, 0.945, 0.945 and 0.955, respectively. The total reliability coefficient for the preference questionnaire is 0.985.

With regard to academic achievement, pre-test and post-test, paper-&-pencil examinations are applied in this study. Eight items of the memorised type and seven items of the non-memorised type are included in these two tests. However, only eight items are the same in both of the pre-test and post-test and the other seven items are different but of the same level of difficulty which is identified by four subject teacher from four high schools and one director of Teaching Affairs.

IV. Results and discussion

A series of statistical test analyses are conducted to examine three types of students' academic achievement. In order to examine the effect of the different teaching approaches on academic achievement, a dependent (repeated measure) t-test is applied. All students' academic achievement in three AR groups is tested at the beginning of (pre-test) and at the end of (post-test) learning the same unit in high school Science by the paper-and-pencil methods. Students' pre-test and post-test are treated as matched variables. Table 1 revealed that all students' academic achievement in three AR groups progresses significantly after the use of the AR learning system. It indicated that the AR learning system did have a positive effect on their academic achievement progress for all three AR groups.

Table 1 A dependent (repeated measures) t-test value for the students' academic achievement in three AR groups

In the analysis of the differences in three types of students' academic achievement, the dependent variable is post-test and pre-test is set as the covariate. On average, the means of academic achievement in AR-Jump, AR-Stretch, and AR-Box are 7.208 (SE = 0.207), 7.603 (SE = 0.205), and 7.145 (SE = 0.207), respectively. However, regardless of the types of physical activity, there is no significant difference among all three AR groups of academic achievement. It revealed that the AR learning system did have the positive effect on their academic achievement progress regardless the types of AR physical activity they engaged in.

In term of students' preferences toward the AR learning environment for the three AR groups, on average, students showed stronger preferences towards the AR learning environments (an average score of 3.71 in Group AR-Jump, of 3.57 in Group AR-Stretch, and of 3.65 in Group AR-Box, respectively, in the 1-5 Likert scales). Table 2 shows the correlations between students' academic achievement and their responses on the preferences towards the AR learning environments.

Table 2 Correlations between students' academic achievement and preferences

The results indicated that, in the Group AR-Box, students' TAA is positively and significantly correlated with two preference subscales: the Ease of Use (EU) subscale and the Challenge (CH) subscale (p < 0.05 and P < 0.005 respectively). In other words, students obtaining higher academic achievement scores tended to prefer AR learning environments where they could operate the AR learning system easily in the technical aspect and they feel the content if the AR learning is challenging enough for them.

In Figure 3, the reference line 'Mean' is the mean of students' academic achievement in Group AR-Boxing. The plot shows that most data converge on the area between reference lines 'Upper Slope' and 'Lower Slope' with very few discrete data. Academic achievement and preference-challenge in Group AR-Boxing has linear positive correlation.

Figure 3

Correlation plot between academic achievement (AA) and Preference-challenge (CH). AA and CH in Group AR-Boxing has linear positive correlation as most data converge between 'Upper Slope' and 'Lower Slope'.

However, for the other two types of AR groups, there is no significant result between students' academic achievement and their preference response on AR learning environments.

V. Conclusions

This study has developed an augmented reality learning system with effective functions of three types of physical activity as well as plentiful learning material to help high school Science learning. An evaluation on academic achievement and students' preferences toward the AR learning environment is implemented. There are some exciting results found in this study.

Firstly, the results indicated that students, in all three types of physical activity within the use of the AR learning system, have significantly positive progress in their academic achievement between the beginning and the end of their studies.

Secondly, regardless to the types of AR physical activity students engaged in, they all obtained the equally positive effect on their academic achievement progress.

Finally, in terms of students' preferences toward the AR learning environment for the three AR groups, students showed stronger preferences (all means > 3) towards the AR learning environments including all seven subscales, Ease of Use, Challenge, Rewards, Situated Learning, Collaboration, Competition and Movement. This study also found that students engaged in the AR Boxing physical activity and obtaining higher academic achievement significantly tended to be in favor of the easy use and of the challenging content in AR learning environment. Further, it is also revealed that students' preference toward the challenging content in the system and their academic achievement significantly had the strongest positive correlation.

The results of the preference evaluation on the AR learning environment provided a promising and effective way for those who want to do learning and exercising at the same time within the use of body languages in an AR learning environment and also a useful reference for those whom are going to develop their own AR learning system.

Authors' Information

Kuei-Fang Hsiao, Associate Professor of the Department of Information Management at Ming Chuan University, Taiwan, has received the B.Sc. degree from National Taiwan University, Taiwan in 1991. She travelled to the United Kingdom for further challenges and research in 'Communication and Mass Media' of Education Faculty and completed her successful distinct M.Ed. from University of Manchester, UK, in 1993. She, then were encouraged to carry out her exciting research project of 'Investigation in Information Technology in Education' who successfully completed with commented Ph.D. in 1998. Since then, her research activities and interests are mainly for using the computer technology for human interactions, education and psychological treatments. Her current research interest is in computing in virtual reality, augmented reality, and AR-sensor technologies for variety of applications in education and healthcare industries.

Professor Habib F. Rashvand, Editor-in-Chief, and Director of Advanced Communications (Adcoms, University of Warwick): Following his distinct engineering qualifications at the University of Tehran in the early 1970s in association with the University, PTT, NTT, KTT and Japanese Industries, Professor Rashvand headed an international project for building the national Telecom Research Centre (ITRC). His academic positions encompass the University of Tehran, University of Zambia, University Coventry, Open University and German Universities of Magdeburg. As the Editor-in-Chief, he managed to change face of the IET Communications Journal and extending into IET-IFS, IET-WSS and with further extension to IET-NET. Under the freedom of Adcoms his collaborative academic activities generated some 50 research papers, keynote speeches and other materials in recent years.