Collaborative design of software applications: the role of users

Abstract

Drawing on a 1-year application design, implementation and evaluation experience, this paper examines how engaging users in the early design phases of a software application is tightly bound to the success of that application in use. Through the comparison between two different approaches to collaborative application design (namely, user-centered vs participatory), we reveal how sensitivity to the role that users may play during that collaborative practice rebounds to a good level of user satisfaction during the evaluation process. Our paper also contributes to conversations and reflections on the differences between those two design approaches, while providing evidences that the participatory approach may better sensitize designers to issues of users’ satisfaction. We finally offer our study as a resource and a methodology for recognizing and understanding the role of active users during a process of development of a software application.

Introduction

The role of users in design and creation processes of computational artifacts is at the basis of many HCI methodologies. In the evolution of such methodologies, the role of the user is becoming more active, being the main character in several steps of the most commonly used approaches for the development and release of applications and tools. We are witnessing at a wide diffusion of many initiatives of open innovation and co-creation in different fields, where different kind of organizations are gradually moving away from traditional design models, becoming more inclusive in their approaches to design and development. Both co-creation and open innovation design techniques are based on a strong involvement of users in the design process [1]. While open innovation suggests active collaboration among different organizations and the sharing of intellectual property, co-creation involves users of a product or service in exchanging their knowledge and resources with the aim to deliver a personalized experience. Co-creation relates more specifically to the relationship between an organization and a defined group of users, who are engaged directly by involving them in the creation of a product or a service, i.e. in its design and in its development processes. Definitions of open innovation and co-creation can be found in [2, 3].

Focusing on the co-creation of software applications, there are different HCI design methodologies involving users that can be applied, such as user-centered design and participatory design. The User-centered design approach was defined more than 30 years ago [4] and it consists in designing and developing applications or products where a team of designers focuses on users’ needs in an iterative way. The team members together plan, create, and develop a project, focusing mainly on design activities and processes. While having the user at the center of the process, a user-centered design can be done without any actual participation of real users, who can be virtualized during the whole design/development/evaluation cycle (modelling stakeholders as personas, or simulating them) [5]. In few words, the user-centered design process can be conducted without assigning to users any active role, but designers are focused on the application being designed, looking for ways to ensure that it meets the needs of the user.

A structured and official definition of participatory design came into the scene later, in 1993 [6]: this design methodology consists in letting the users actively contribute to the design and to the content development process. The participatory design involves users in the process as co-designers, in a more in-depth way. In fact, they can propose and generate design issues, functionalities, services of the application being designed. Hence, participatory design supports different ways of planning and thinking, making applications and products more responsive to human needs. It basically brings together designers, developers, and users to think and define the contexts of use at the design phase [7]. Summarizing, in participatory design experiences, the roles of designers and developers blur, while the role of users becomes a more prominent component of the process.

Differences and similarities of these two approaches are still debated, as witnessed by discussions and topics in the most commonly used social networking sites for scientists and researchers (such as Researchgate.net [8] and Academia.edu [9]). In particular, intersections and borders of these approaches, together with their application, are still arising interest in the academia. Some interesting examples are reported in [10, 11]. Indeed, one of the most interesting interpretation is that participatory design is user-centered, when it focuses on the interest and on the satisfaction of the end-users [12].

In this context, our aim is to investigate if a more deeply engagement of users (reached through a participatory design process) corresponds to a higher level of satisfaction in users of the designed application, when compared with the satisfaction level of the users who are exploiting an application resulting from a more traditional process, based on a user-centered design approach.

In order to reach this goal, we have applied the user-centered design and participatory design approaches to the development of two applications, where end-users have been engaged. Then, we have involved users in the evaluation of the two resulting prototypes, with the aim of measuring their satisfaction in using them, and of discussing emerged differences. In particular, here we present two experiences conducted at the University of Bologna, where students collaborated in the design of two applications thought to be used by exploiting mobile devices:

• 1. An application designed to support foreign students (Erasmus, exchange, overseas, International, mobility students, etc.) in their daily activities, getting in touch with local language and culture, which enhances their integration and their interaction with the country hosting them.

• 2. An application designed to support high school students while approaching their undergraduate programme and become a freshman.

While the first application was designed with a user-centered approach (users were not involved in defining goals, functionalities, and services of the application, they were engaged just in a latter design phase, by means of questionnaires, focus groups, and experience prototyping activities), the second one was based on a participatory approach (users were asked to propose ideas in specific co-creation sessions). After the development of two prototype applications, we have conducted some tests with stakeholders, letting them evaluate the application. Then we asked them to fill a survey, with the aim of comparing users’ satisfaction and of evaluating how the resulting prototypes meet users’ expectations and needs. Moreover, we want to discuss about differences in expectations and satisfaction when users were involved in both the design and in the evaluation processes, and when they were involved only in the final evaluation phase. In particular, we aim to observe if there are differences between the satisfaction levels reported by these two groups of users. The results we have obtained show how the prototype resulting from the participatory design corresponds to a higher level of users’ satisfaction and how it is more compliant with users’ expectations than the application coming from the user-centered design approach.

The remainder of the paper is organized as follows. “Research questions and methodology” section reports our research questions and the methodology we have applied. “User-centered design: the ILocalApp experience” section presents the experience based on the user centered design approach, while “Participatory design: the AlmaOrienta App experience” section describes the process based on the participatory design. “On comparing application functionalities and services” section illustrates some common elements and functionalities emerged by the two design processes, highlighting main differences. A final evaluation phase conducted with end users is reported in “On evaluating the design processes and their results” section, detailing users’ satisfaction and assessing how the resulting prototypes meet users’ expectation. Finally, “Conclusion” section concludes the paper illustrating main findings and further work.

Research questions and methodology

In this section, we report the research questions (RQ) we have formed and the methodology we have applied with the aim of prove them:

• RQ1: Users’ satisfaction is proportional to real users’ engagement in the design process of an application.

• RQ2: Applications resulting from participatory design approach better meets users’ expectation than the ones resulting from user-centered design.

In order to answer to our research questions, we have taken the chance of applying different methodologies in the process of designing two applications devoted to meet different needs of users at the University of Bologna, with different levels of users’ engagement in such a process.

In the first case, we have applied a user-centered design approach while defining goals, functionalities, services, user interface and interaction for an application devoted to support foreign students hosted by a university while they are experiencing an exchange programme. In particular, in this experience, a team of designers has defined the goals, the services and the functionalities offered by the application, on the basis of a user model, aiming to meet the needs of the user. In a second phase of the design process, real stakeholders (i.e., International students, students of mobility programmes, past and future Erasmus students) have been engaged by filling a questionnaire and by taking part in focus groups and experience prototyping activities, as detailed in “User-centered design: the ILocalApp experience” section.

In the second case, we have applied a participatory design approach to the development of an application devoted to support high school students while they are choosing their undergraduate programme and while they are becoming freshmen. In both cases, we have involved target users in the design processes. In this context, the experience was mainly based on participatory design sessions, where high school students have taken part, by proposing ideas and by sharing requirements and desiderata, and on a development contest, where undergraduate degree proposed their solutions and prototypes, as illustrated in “Participatory design: the AlmaOrienta App experience” section.

After the design phase, we have developed the applications (thought to be used by means of mobile devices), whose main features are presented in “On comparing application functionalities and services” section. Then, we have involved target users in testing our prototypes and we have collected feedbacks by letting them fill a questionnaire. Some of the users involved in this last phase were originally involved in the design process, while some others did not take part to these initial activities, letting us discuss about how these latter users show a higher level of satisfaction (as described in “On evaluating the design processes and their results” section).

User-centered design: the ILocalApp experience

The ILocalApp project (incidentally learning other cultures and languages through an APP) is a 3-year Erasmus+ KA2 project, carried out by a transnational consortium (University of Bologna, Italy; the Adam Mickiewicz University, Poland; University of Lapland, Finland; Centre for Social Studies, Portugal). Its main goal is to design and develop a mobile application for incidental learning [13, 14] of four cultures and languages: Finnish, Italian, Polish, and Portuguese [15]. ILocalApp aims to let its users acquire language and culture, thanks to learners’ participation in an informal way [16]. Learners would be able to wander around the city while using the application to enjoy language and culture geolocalized contents in a context-aware way. Thus, mobility students (e.g. Erasmus, exchange, overseas, International, mobility students, etc.) would be supported in their daily activities, getting in touch with local language and culture in a simple and innovative way, which enhances their integration and their interaction with the country hosting them [17].

The project has been based on a user-centered design approach, with the aim of designing and developing a mobile app, which takes into account the user and his/her needs, preferences, abilities, since the very first step (as described in [18]). In order to reach this goal, the project has carried out many activities, such as focus groups [19] and experience prototyping [20], by involving undergraduate and international students at the four partners sides, after an initial phase during which designers have defined goals, features, and services of the application. The results coming from such activities have driven the development. In particular, students were asked to participate by filling a questionnaire (as described in “Phase one: questionnaire” section), by attending focus groups (as presented in “Phase two: focus groups” section), and by evaluating a preliminary prototype through an experience prototyping activity (as reported in “Phase three: experience prototyping” section).

Phase one: questionnaire

An online survey has been launched with the aim of finding out students’ views on language learning apps and the role of apps and mobile devices in their everyday life. The survey was launched from February 16th, 2016 to March 28th, 2016 and it had reached altogether 2350 answers all over Europe.

The survey was structured in 20 questions, grouped in three parts, devoted to identify and collect information about: (i) culture areas of interest, views on using language and culture apps; (ii) usage of apps and mobile devices (e.g., operating system on the smart phone, most commonly used apps, frequency of use, etc.); (iii) personal details (e.g., age, gender, first language, etc.). The most relevant results coming from such a survey were exploited to better identify the nine categories of contents, the issues motivating users of language and culture apps (short and useful learning moments, interactive, tracking progress, efficient), and the most commonly used language and culture apps (such as Duolingo and Memrise). 53.5% of the users declared they usually install apps with offline maps and/or city guides or similar when they are planning a trip. Taking into account personal details, 42.9% are aged 19–22 years, while 37% are between 23 and 26 years, confirming the targeted audience of the project, which was constituted by young and emerging adults. Finally, the targeted groups of users have been also confirmed by how the users have identified themselves: students who are planning to go on an exchange (39.4%); users who currently are or have previously been exchange students (33.1%); international degree students (12.9%).

Phase two: focus groups

In April 2016 focus groups were held in all the ILocalApp institutions with the aim of getting feedback and comments from participants on the functionalities, the interactions, and the flow of the application that we were developing (in terms of how and when offering contents and activities). In particular, the goal of the focus groups was progressing in the definition and the design of the application functionalities and services. Table 1 summarizes the students involved from the 4 partners in the focus groups (as well as in the experience prototyping activities, described in “Phase three: experience prototyping” section). Participants were selected among the identified target groups (mainly international students, past, present and future Erasmus students, exchange students, overseas students, etc.).

Table 1 Focus groups and experience prototyping participants

The discussion of all the sessions was structured in five main topics, as agreed by all the involved institutions as detailed in the following. Some mockups have been exploited so as to facilitate the discussion among the participants. The discussion brought concrete suggestions and hints from the participants who actively discussed the proposed topics during the sessions. Reporting all the collected feedback and comments coming from the 4 partners, we can summarize as follows the focus groups results:

• 1. Content organization The importance of having information related to the user’s position emerged as a common element in all the conducted sessions. In particular, it has been identified as strategic having an integration of mobile/web mapping services (such as Google Maps) in the application. Information about public transportation, events, and points of interest have been defined as interesting, as well as information about useful services (e.g. pharmacies, hospitals, citizenship/immigrant services) in the nearby. Particularly appreciated would be having language tips of immediate use, context-related, “how-to” phrases, including survival dictionaries and handy expressions. Cultural issues have been reported as crucial, in terms of places, monuments, cultural heritage, but also as a concrete way of living.

• 2. Application usage The usage of the application before going to the place has been identified as very useful to start to know the language and the place, to have communication tips, and to get used to the application. Obviously, the emphasis is recognized during the stay, when the application is mostly important.

• 3. Usage frequency Among all the institutions, participants marked that the application should not force the users, even if some notifications or services could be exploited as motivation (eventually by means of game-like elements [21, 22]). It is important to let the user free to decide and modify the level of notification. The rhythm of usage may vary (ideally, it could be higher at the beginning of the stay), without forcing the user to do a minimum of activities.

• 4. Interface, icons vs language of instruction, presentation video, progress check Participants declared that they would prefer a user interface based on a combination of icons and language of instruction, with priority to images. The user should have the possibility to change the language (English in principle, but then they could switch to the local language). Game-like elements have been recognized as useful to stimulate and to monitor progress.

• 5. Interaction with other learners and with native speakers This last topic was the one with most opposite and contrasting results. In particular, two complete opposite positions emerged from all the focus group sessions. On the one side, participants in Bologna and in Coimbra considered as relevant the interaction with other users (both native speakers and other learners), eventually integrating or linking other existing networks and social media in the application. On the other side, participants in Poznan and Rovaniemi claimed that there is no need to add functionalities and services to support interaction, since there are plenty of means, tools, channels, and strategies with this aim that can be exploited.

Phase three: experience prototyping

In May 2016, experience prototyping sessions were held in the ILOCALAPP institutions, with the aim of getting feedback and comments from participants on the functionalities, the interactions, and the flow of the application that we were developing, by means of a realistic simulation in a real world context, by exploiting the “Think aloud protocol” [23]. Table 1 summarizes the students involved in the experience prototyping activities. In particular, the goals of the experience prototyping sessions were: (i) understanding users and their experiences with the app prototype in a real world context; (ii) understanding users’ needs and users’ point of view in a real user environment.

To reach these goals, we have identified three tasks the participants had to complete during the experience prototyping session. The tasks are the same ones for all the ILOCALAPP partners: (i) buy a bus ticket; (ii) visit a monument/museum/library; (iii) eat a typical food. Starting from some “common” mockups, we have customized them with specific multimedia content (in terms of pictures and texts) for each partner, as prototype models (see Fig. 1 showing the prototype preparation phase in Bologna). Figure 2 shows a picture taken during experience prototyping sessions conducted in Bologna.

Fig. 1

A picture taken during the prototype preparation phase in Bologna

Fig. 2

A picture taken during experience prototyping sessions in Bologna

We have analyzed the notes and the reports provided by all the partners involved in such an experience, and we have identified the main common remarks about the following items:

• Navigation among the contents and the services It plays a strategic role. Navigation tools should be adequately provided to come back to the previous content/menu/interface, to come back to the home page of the application, and to move across different services/content/functionalities related to the same content (i.e. language tips, cultural and historical information, practical information, games, external links, etc.).

• Interface, layout and their configuration/personalization The participants have shown opposite positions about some layout and interface elements, such as font size. This means that interface personalization is needed [24, 25]. The application should offer a set of configurations to let the user choose his/her preferred combination of layout and typographical aspects, meeting his/her needs, so as to improve the user’s experience, also in terms of learning experience [26].

• External links and social network connections External links and connections to social networks are considered important. In particular, participants expressed interest in having: (i) external/official information related to the topic; (ii) a way to share information/pictures/comments; (iii) exploiting already existing social networks.

• Practical and cultural content A good balance between historical/cultural content and practical information should be found. Designers and developers have to pay attention in providing not only practical information or not only historical/cultural information.

• Audio content Audios are needed, in particular for the language tips, with the aim of effectively support users in their daily activities and in learning the language.

These results confirmed some issues already emerged during the focus groups and let the designers and developers better understand the direction of the final design of the application and better identify the direction of further steps in its development.

Participatory design: the AlmaOrienta App experience

To promote its programs, the University of Bologna organizes a spring college fair, called AlmaOrienta. The fair is held in the Bologna Exhibition Center (Bologna Fiere) in 3 exhibition halls that covers more than 25,000 mq. Every year about 30,000 high school students visit AlmaOrienta with the aim of finding support in deciding about their future opportunities at the University of Bologna. The complexity of the programme catalogue and the admission procedures together with the wideness of the AlmaOrienta event drove the University to provide students with a mobile application to support them while approaching their undergraduate programme and become a freshman. Thinking to the digital native generation who is approaching the university in the next 2 years, we decided to use co-creation in designing the new application, involving students in the design process. This goal is generally challenging, but it is much more complex in the specific situation, due to difficulties in identifying a community of users to be involved in co-creation. On the one hand the target community, students who are attending the last 2 years of high school have the need to be supported during and after the fair, while deciding about their future studies, but they are not enough aware of the complexity of the programmes catalogue to completely design a similar solution. On the other hand, the community of university students have the appropriate level of expertise about how the university works, but they are not necessarily aware of the real needs of the next generation of students. To overcome this situation, we decided to involve both the communities of students in the application design, using a combination of two co-design strategies:

• A participatory design phase involving the target group of high school students in providing proposals. While this group of users have a clear understanding of their own needs in terms of support in deciding and starting college studies, they have a very partial view of the complexity of the programme catalogue offered by the University. The participatory design phase took place during AlmaOrienta 2017, in March 2017.

• A contest (with a call for ideas and a prototypes’ development track), involving University of Bologna undergraduate students in structuring, improving and implementing proposals arose during the participatory design phase. Students enrolled in the University know difficulties they got in deciding and, at the same time, they have a reasonable perception of the complexity of the programme catalogue offered by the University. The contest took place in spring 2017 and it was completed during a final contest session in May 2017.

The combination of the two phases produced a wide set of proposals of different on line support services, thought to be used during the AlmaOrienta fair and after (while high school students are deciding about their future university studies).

Phase one: participatory design

During the 2017 edition of AlmaOrienta, we conducted several participatory design sessions, involving high school students who were attending the fair in design activities under the moderation of colleagues from the Computer Science and Engineering department. At the end of the 2 days of fair, we have globally collected 45 project ideas, proposed by a community of more than 200 high school students. Figure 3 shows one of the mockup proposed by one team composed by three of those students. Proposals focus on 8 main topics:

Fig. 3

A mockup created during phase one: participatory design (in Italian language)

• a. Programmes catalogue Almost all the proposals asked for a structured presentation of the catalogue and some support tool to better understand differences among programmes.

• b. Identity The collective value of the University of Bologna is expressed though clear and strong identity message, including a clear visual identity.

• c. Tests a significant part of the University of Bologna programmes has an assessment admissions test. Students’ designs underlined the need to be supported in studying for each specific admission test and in managing the application for the test and the following application for the selected programme.

• d. Fair AlmaOrienta fair many design proposed by students include services to present, explain and navigate the AlmaOrienta fair itself, together with open days and all the events organized by the University and its departments, schools, and programmes.

• e. Places The University of Bologna has a multi-campus structure, based on 5 campuses in 5 different cities. The Bologna Campus itself is structured in 9 different areas inside and outside the city. Projects proposed to support students in physical orienteering in the wide university space, including information about the city (e.g., transportation, dormitory and other residential facilities, lifestyle).

• f. Job traditionally programme catalogues aim to support students in understanding the structure of programmes, as they are expected to run for the given year. The programme and unit catalogues are organised by school and level (first cycle, second cycle, single cycle) without considering that different programmes can lead to similar competences and consequently to similar jobs. Proposals arose during the participatory design mainly focus on what jobs or professional activity users can do once graduated.

• g. International although students involved in the participatory design come from cities nearby Bologna, some proposal underlined the international dimension of the University and proposed new services to international students or to students who may apply exchange programmes (i.e. Erasmus, Overseas, etc.).

• h. Communication style most of the participatory design projects underlined the need of a clearer and direct communication style. They explicitly refer to chat, messaging, forums and other social communication tools to contact counsellors, tutors, professors, and also other students and alumni.

Phase two: mobile application development contest

The second design phase was organized as an on line (remote) application design contest, named AlmaContest and open to the whole community of students of the University of Bologna. Almacontest was structured in two tracks in order to exploit:

• A. Ideas Track A was devoted to collect project ideas to be used in the development of the university fair app.

• B. Prototypes Track B asked students to develop a prototype to be used as a support for high school students.

Students applied to AlmaContest in teams, ranging from 2 to 5 members, selecting one of the two tracks. For each track, a monetary reward was available for the best 3 projects of the track. A jury of experts, including ICT professors, design specialists and college counselling managers, was nominated to select the best projects. Globally 59 students applied for the AlmaContest, organized in 22 teams, 14 of them for the track A (ideas) and 8 for the track B (prototypes). The best 5 projects of each track were involved in a contest final session, where each selected group gave a speech (as a 5 min presentation with slides) to the jury. The best 3 projects for each track were selected and announced as winners immediately after the speech session. Figure 4 shows a picture taken during 1 of the 5 min presentation, in the speech session. Table 2 reassumes characteristics and covered topics of the 6 winner projects.

Fig. 4

UniversiTalk presentation during phase two mobile application development contest

Table 2 Characteristics and covered topics of the six winner projects

Although the most part of AlmaContest projects, including the winner ones, refer to topics underlined during the participatory design phase, some significant aspects arose from this second design phase. Projects introduced both creative and implementation novelties. Some very new services were proposed such as (i) using ebeacons to support geo-localization inside AlmaOrienta fair and track users’ while they explore the fair [27]; (ii) using gamification so as to keep users engaged both during the fair and after (project 5, Universitalk); (iii) using a chatbot to answer students’ questions; (iv) computing similarity among programmes by using a clusterization algorithm (project 2, AlmaMatch).

On comparing application functionalities and services

In order to compare the results emerged from the two design experiences in terms of application functionalities and services, in this section we present the most interesting issues emerged from applying these two design approaches, highlighting differences and similarities:

• Geolocalized information and services Both the applied design approaches let emerge or reinforces the need of exploiting data and services on the basis of their geolocations and of users’ position.

• Game-like elements and gamification The use of gaming factors emerged as significant and strategic from both the applied design approaches. Mainly they could be applied with the aim of involving users and supporting them in conducting activities related to the application [28].

• Communications paradigms Different needs in terms of communication emerged. On the one hand, from the participatory design experience it was strong the need of a synchronous and direct mechanism of communications. Users involved in that experience suggested the use of chats as well as chatbots. They expressed the need of direct communications among people with different roles: from their peers, to alumni, from professors, to administrative staff. On the other hand, a direct form of communication with other users is not emerged as so crucial in the user-centered design experience.

• Notifications The use of push notifications is recognized as essential in both the experiences. In user-centered design approach, it should be exploited with the aim of letting the user be aware about the presence of points of Interest in the nearby, while in the participatory design approach it should be used in order to report deadlines and training activities.

• Personalization/preferences/settings The need of adapted information and services, availability of setting preferences and personalized configurations is considered as crucial by the users involved in both the experiences. The two adopted design processes let this need emerge as one of the strongest requirements.

• Incidental learning mechanisms This emerged as a strong need in the user-centered design experience, while it is not relevant in the participatory design obtained results.

• Interaction and social networks integrations The need of social networks integration emerged from both the experiences. It has been reported as a more important need in the participatory design side, than in the user-centered design experience. In particular, users showed interested in the integration of communication systems similar to Telegram.

Some of these emerged functionalities and elements are common at the two experiences, while some others differ. In particular, it is interesting the role and the need of communication among users, which emerged in a totally different way in the two design approaches.

On evaluating the design processes and their results

After the development of the two prototypes, before launching the applications, we have involved the users in testing them, collecting their feedbacks and comments by means of a survey. In particular, we involved users who have previously taken part in the design activities (groups A) and users who did not, who have neither bias nor specific expectations about the applications and their purposes (groups B).

The survey was structured in three parts:

• 1. Personal details (e.g., age, gender, current position, etc.);

• 2. Evaluation of the application, in terms of user’s satisfaction, ease of use, user’s engagement;

• 3. Additional open comments.

An additional fourth part was provided to users of groups A, who were involved in the design activities, with the aim of evaluating their appreciation about their involvement in the design process. We use the Likert scale [29] approach in ranking the second part questions based on a 5 values symmetric agree disagree scale, in order of strength. In particular, the adopted scale was as follows: strongly agree, agree, neutral, disagree, strongly disagree. Moreover, we have included an open question, to let the users add some details about their answers. Questions of this survey part are reported in Table 3.

Table 3 Part 2 survey questions (and answer models)

Moreover, we gave participants the possibility to provide personal comments in the third part of the survey. We have provided the questionnaires to 2 different groups of users for each design experience, as follows:

• 1.a Erasmus, exchange, International, mobility students, who took part at the focus groups and experience prototyping activities in Bologna, during the user-centered design phase (10 students).

• 1.b Erasmus, exchange, International, mobility students, who were not involved in the user-centered design phase (11 students).

• 2.a High School students, who took part at the participatory design activities (13 students).

• 2.b High School students, who were not involved in the participatory design phase (12 students).

Obtained data were analyzed from both a quantitative viewpoint and from a qualitative perspective, obtaining interesting results, as here reported.

On a quantitative perspective, taking into account users’ satisfaction about the application, a first element of interest is the fact that all the users enjoyed the app (Question 1), providing positive feedbacks (“strongly agree” and “agree”), claiming they would use it (Question 4) in choosing their undergraduate programme or in getting confident with his/her hosting university during an exchanging programme. Moreover, only 6 users (13.04%) claim that the app does not meet their expectations (in particular, users belonging to group 1.a, 1.b, and 2.a) and only 2 users (4.34%) declare that the app is boring (both were involved in the participatory design experience, 1 belonging to group 2.a and 1 belonging to group 2.b). Aggregated data reporting users’ answers about questions related to users’ satisfaction are shown in Fig. 5.

Fig. 5

Users’ appreciation about the application (Question 1), users’ expectation about the application (Question 2), users’ perception about application boredom (Question 3), users’ willing in using the application (Question 4)

Figure 6 depicts answers users have given about issues related to the ease of use of the applications. In particular, only 5 users (10.87%) claim that using the application has not been simple (Question 5) and that they have faced some troubles while accomplishing some tasks (Question 6), while 7 users (15.22%) do not appreciate the apps layout and interface (Question 8). Finally, only 5 users (10.87%) answered that they need a consistent amount of time to learn how to use the app, stating its ease of use.

Fig. 6

Users’ perception about the ease of use of the app (Question 5), about difficulties in completing tasks using the app (Question 6), users’ perception about time needed to learn how to use the app (Question 7), users’ appreciation about the app interface (Question 8)

On the open comments in part 3, a user (group 2.b) wrote: “I will surely use this application to choose my undergraduate program. It is better than the official web site, clearer and more focused.” While another one (group 1.b) commented: “The application could be useful not only for foreign students but also for offsite students and for freshmen who are not used to live and study in Bologna.”

We have observed general better response from users involved in the participatory design experience, with a more in-depth involvement of the users, who were asked to provide their own ideas. Table 4 compares the percentage of positive feedbacks for the two prototypes, focusing on the users’ satisfaction. In particular, we have taken into account “strongly agree” and “agree” answers for all the questions, except for Question 3, which is a control test question regarding users’ satisfaction (hence for this item we report the percentage related to “disagree” and “strongly disagree” answers). Data shown in Table 4 confirms that the more target users are involved in the design process, and the more end users appreciate using the resulting prototypes.

Table 4 Positive feedback to questions related to users’ satisfaction

Concluding, the results here reported confirm the RQ1 research question we have formed in “Research questions and methodology” section: “Users’ satisfaction is proportional to real users’ engagement in the design process of an application”.

In order to verify the RQ2 research question (“Applications resulting from participatory design approach better meets users’ expectation than the ones resulting from user-centered design”), we reported users’ answered divided in A and B groups to Question 2 (“The application meets my expectation”). In this sense, we have observed a better response from users belonging to the B groups, in other words from the users who were not involved in the design processes, as shown in Table 5, which reports details about the answers of the 4 groups of users. These data show that groups B responds with a higher percentage of positive feedback (“strongly agree” and “agree” answers) than groups A of users. In particular, taking into account the user-centered design experience, group 1.b gave 63% of positive feedback, while group 1.a gave 50%. Participatory design users group A gave 61% of positive feedback, while group B 83%. Similar results can be observed in the answers of the most part of the other questions. This can be due to a bias: groups A of users were involved in the design phase, and, during these activities, they have built specific expectations of the mobile app, while the final prototype could have met only parts of these expectations. Again, it is evident that the participatory design approach have obtained more positive feedbacks than the user-centered design one, both from users previously involved in the design phases and from users who did not take part in those activities.

Table 5 Detailed answer to item 2 in the questionnaires

To assess the significance of the experimental results from a statistical point of view, we carried out an additional statistical analysis. In particular, for three items in the questionnaires (provided to the four groups of the users involved) we defined a null hypothesis and calculated the corresponding p-value using the standard two-tailed formula for the one sample t-test. As each item in the questionnaire can be responded with a value in the interval [− 2,2], we assumed 0 as the expected average value (accordingly with the related null hypothesis) and considered a confidence level as great as the 95%. Hence, the level of significance was set to 0.05. From an analysis of Table 6, it is easy to verify that each of these p-values calculated were smaller than the 0.05, thus providing a statistical significance to our experiments and allowing us to reject each null hypothesis.

Table 6 One sample t-test analysis and relative measurement for three questions

Conclusion

Participatory design is based on a more deeply engagement of target users in the design phases if compared with the user-centered design approach. This paper aims to prove that a stronger involvement of users in the design phases corresponds to a higher level of users’ satisfaction while using the resulting application. In order to demonstrate it, here we present two experiences of design process, where user-centered and participatory design approaches were applied. In particular, target users were involved in different ways in the design phases of two applications, one devoted to support foreign students during their exchange programme and one devoted to support high school students while they are choosing their undergraduate programme. The paper describes how we have conducted the design phases and how we have involved their target users in the process. After the development of the prototypes, we went back to target users, involving them in an evaluation phase and asking them to provide feedbacks by means of questionnaire. In particular, at this stage, we have involved users who took part at the design phase and users who were not involved at that time, hence users who have no expectation and no idea about the applications purposes and goals. Results we have obtained confirm the two research questions we have formed:

• RQ1: Users’ satisfaction is proportional to real users’ engagement in the design process of an application.

• RQ2: Applications resulting from participatory design approach better meets users’ expectation than the ones resulting from user-centered design.

Users of the application resulting from the participation design obtained a better evaluation in terms of meeting users’ expectation, compared with the user-centered designed one, confirming RQ2 research question. Moreover, users claimed they are satisfied by the two prototypes, but the application resulting from the participatory design approach got higher scores from users in terms of their satisfaction, confirming our RQ1 research question, hence letting emerge a relationship, a proportion, between target users’ involvement in the design process and users’ satisfaction in using the application prototype.