Skip to main content
Download PDF

Qualitative usability feature selection with ranking: a novel approach for ranking the identified usability problematic attributes for academic websites using data-mining techniques

Human-centric Computing and Information Sciences volume 7, Article number: 29 (2017) Cite this article

Abstract

Objective

The aim of this study is to identify common usability problematic patterns that belong to top-50 academic websites as a whole and then ranking of these identified usability problems is also provided.

Methods

In this study, a novel approach is proposed that is based upon the integration of conventional usability testing and heuristic evaluation with data-mining knowledge discovery process. An experiment is conducted to evaluate ISO 9241-151 guidelines under 16-different categories by hundred participants who are frequent users of academic websites. After evaluation, the qualitative usability data is collected and different data-mining techniques i.e. association rule and decision tree are applied to recognize fully functional and problematic usability attributes. Identified problematic attributes represent common usability problems patterns related to academic websites from the qualitative viewpoint only. This study further prioritizes these problematic attributes by using the ranking algorithm that represents the order in which usability issues must be resolved.

Results

In this study, 16-different categories are considered for usability evaluation of academic websites. The results show that no issues are identified in two-categories i.e. {Headings_Titles_Labels and The Home_Page}. In Scrolling and Paging category, horizontal scrolling is identified as a major issue whereas, in Internationalization category, the users do not identify supported languages on most of the academic websites. Users do not find websites to be highly secured under Security category. Our findings investigate that most of the issues are found in Search and Social Media categories. Furthermore, users easily locate 50.53% guidelines on websites as fully functional whereas, 49.46% of characteristics are considered as problematic usability features that are not functional on the academic website as a whole.

Conclusions

Identification of common usability problems at an early stage can lower substantially the development efforts in cost and time. Software developers can restrain from these potential usability problems during the development of novel systems under the same context. Providing appropriate solutions for these problems can become valuable in software development. The proposed approach concludes that conventional usability evaluation methods can go beyond just than testing of systems. The study is a milestone towards identification and prioritizing problematic usability features for academic websites and helps in providing the wholistic approach of usability problematic patterns for web-domain.

Introduction

The concept of usability has emerged from the term user-friendly. Software usability is defined as the ease of use of software [1]. IEEE Std.610.12 [2] explains usability as “the ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component”. ISO 9241-11 [3] defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, satisfaction, and efficiency in a specified context of use”. “Context of use” means the description of actual conditions under which an interactive system is being evaluated. Context analysis is generally required for conducting research on software usability [4].

The major challenge in usability engineering is related to usability evaluation methods (UEM). These methods emphasize on evaluating the interfaces of the systems [1, 5, 6]. But these methods appear to be complicated when a huge number of systems belonging to the similar context of use, are evaluated collectively to provide an extensive diagnosis. The prime cause for this is the substantial amount of information that must be handled and conceptualized concurrently. In order to overcome this challenge, these methods must be extended with data-mining knowledge discovery process. A number of research studies have been reviewed that employ data-mining techniques for evaluation of usability [7,8,9,10,11,12]. Moreover, usability evaluation has been done in different domains e.g. mobile [13,14,15,16] and website [17,18,19,20,21]. It has been identified that usability evaluation is extensively performed for web-domain [18]. Therefore, it further motivates us to focus on the assessment of usability in web-domain.

Usability assessment is usually performed through quantitative and qualitative assessments. Quantitative assessment measures the different dimensions of usability and computes the results through a single metric [5, 22]. On the contrary, quantitative computation is not strong enough to measure the overall usability of any system. Therefore, usability professionals conduct qualitative assessments of the system mainly, during the evaluation stage. It is very crucial stage due to the fact that user feedback is considered till the admissible level of usability is attained. Hence, this makes us motivated to focus more on the qualitative evaluation of software usability.

Another major challenge in usability engineering is the detection of the fully functional and problematic usability features for a context of use as a whole. Here, a context of use as a whole refers to the simultaneous analysis of usability issues across different system within the same domain. In order to overcome this challenge, usability testing and heuristic evaluation are performed that emphasize on the “what” over the “how many” questions related to detection of functional and problematic usability features for a context of use.

The objective of the study is the detection of common usability problems in any system in a given context of use. Proposing suitable solutions for these problems can become valuable in usability engineering or for software development in various aspects. Usability professionals can also be benefited by depending upon these identified usability problems in a given context when novel interfaces of systems in similar context are evaluated. Moreover, this contextual information regarding usability problems can assist software developers to restrain from these potential usability problems during the development of any novel systems under the same context.

Further, in the present study, various research gaps related to difficulties in understanding the users and prioritizing fixing the identified problems have also been addressed [23,24,25] and motivated us to focus more on prioritizing fixing the identified usability problems. As the resources are finite, it is essential to prioritize identified usability issues in such a way that can enhance analysis. Especially when evaluating a large number of systems, the prioritization issue directs the evaluation team to focus on what actually matters, saves time, and efforts.

This paper proposes a novel approach of QUFSRc Footnote 1 the process that is based upon the integration of qualitative usability testing, and heuristics evaluation, using data-mining knowledge discovery process (Association rules, decision tree and attribute selection using ranking algorithm). In order to validate the proposed approach, an experiment is performed in which academic websitesFootnote 2 are evaluated to find common fully functional and problematic usability attributes. Problem prioritization is also provided so that high prioritized usability problem should be immediately and carefully handled. In short, a general diagnosis for the context of use for websites is presented by applying QUFSRc approach. This approach uses ISO 9241-151 [26, 27] guidelines and other heuristics belonging to latest technological issues. Every guideline presents usability feature of websites and is evaluated by hundred participants. The ISO 9241-151 and other heuristics guidelines are used as a source for check box list for real end-users as a guide during usability evaluation. Generally, manual data collection is performed that is time-consuming and cumbersome job [8]. So, Google form is used that helps the participants to provide answers to usability guidelines within a minimum time resource.

The remaining sections of this paper are arranged as follows: in "Related work" section, related work is explained. "Brief overview of data-mining techniques used" section describes the brief overview of data-mining techniques that are used. "A proposed approach for qualitative usability feature selection with ranking (QUFSR)" section describes the proposed approach that augments the conventional usability testing, heuristic evaluation and ISO-9241-151 guidelines with data-mining knowledge discovery process. This novel approach determines functional and problematic usability attributes for a context of use as a whole, but mostly from a qualitative viewpoint. "Experimentation and results" section describes experimental results that validate the proposed approach. "Conclusions and future work" section presents the conclusions and future work. This work is based upon some preliminary research work conducted on usability evaluation of early prototypes through association rules [7], and usability testing through data-mining techniques [9, 12].

Related work

For evaluation of any systems, different usability models and methods are proposed [9,10,11,12] with main focus given on usage-centered interface designs. Various studies have provided the state of art of methodologies in usability engineering [1, 6]. Generally, usability testing [1, 6, 14, 28,29,30] and heuristic evaluations [31,32,33] are employed for usability evaluation. Qualitative usability evaluation [8] is conducted to impart usability diagnosis of a given context of use. Some studies have proposed an integrated approach of usability evaluation with different data mining techniques. By using association rule, usability evaluation of early prototypes is conducted [7, 9]. A recommendation model is proposed with a description of a set of association rules that can improve the usability of the system [10]. Besides, decision trees are implemented for conducting the usability evaluation [11]. Few articles have implemented an integrated approach for usability evaluation mainly with knowledge discovery in database process (i.e. association rules and decision rules are implemented) and significant usability problem patterns are identified belonging to homepages of websites [8, 12]. Fuzzy and model-driven development approaches [34,35,36,37,38] are also adopted for usability evaluation of software and websites. Usability evaluation has been significantly adopted in various domains e.g. mobile [13,14,15,16] and website [17,18,19,20,21].

KDD stands for Knowledge discovery in a database. It is an approach to extract hidden and previously unknown information. The extracted information is significantly constructive information that infers from the available data. Generally, KDD normalizes the data from heterogeneous sources and then collects it in data repository. From the data repository, attributes are generally determined. This extraction process in data-mining knowledge discovery [12, 39] is decomposed into four main processes (a) data pre-processing, (b) data-mining, (c) pattern identification and (d) graphical representation. A software platform for data-mining knowledge discovery process generally involves 3-components i.e. data repository, data-mining knowledge discovery engine, query interface [12, 39]. It is important to note that data-mining is the most vital step in data-mining knowledge discovery process since it extracts previously unknown information and identifies hidden patterns of information for evaluation of patterns [39]. Data mining tasks demand the use of robust software platforms. Different free-open source platforms have also been used for conducting data-mining knowledge discovery process [39]. e.g. RapidMiner [40], R [41], Orange [42], and Weka [43, 44]. This paper is based upon some preliminary research work done by Gonzalez et al. [7], on usability evaluation of early prototypes through association rules and usability testing through data-mining techniques [8, 12]. For usability evaluation, Weka workbench is frequently used for applying data-mining techniques. Therefore, in this study, Weka workbench is selected to implement data-mining techniques [39, 44, 45], e.g. association rule, decision tree and attribute selection algorithm. In next sub-section, main characteristics of these three-algorithms are summarized.

Brief overview of data-mining techniques used

Association rule

Association rules data-mining technique helps us in determining different patterns in datasets. Besides, association rule uncovers significant relationships among data-items in a data set [39, 43, 45, 46]. The detection of these relationships among database records are frequently used in different decision-making approaches and for real-world problems, e.g. cross marketing and market basket analysis. The discovery of these relationships may assist decision makers to have strong marketing planning and strategies. Different algorithms have been used for creating association rules from large transactional databases e.g. FPGrowth, Apriori, Filtered Associator and PredictiveApriori etc. [45, 46, 49]. Two fundamental parameters are used to evaluate interestingness of association rules, named confidence and support. In order to reduce computational complexities, thresholds values are also provided for support and confidence parameters. Table 1 represents an example of a database on which association rule mining can be implemented.

Table 1 A sample database with example representing association rule
Full size table

Decision tree

A decision tree is flow-chart-like tree structure that contains internal nodes, branches, and leaves. Every internal node represents a test on an attribute value whereas every branch depicts an outcome of the test. Leaves belong to classes or class distribution [39, 45]. In the decision tree, instances are classified and sorted down from root to leaf nodes. Every node carries a test on attribute whereas every branch belongs to one feasible value for that attribute. For the classification of an unknown sample, attributes values must be examined against the tree. The path is generally followed from root of a tree to leaf node that predicts the class for given sample. There are different algorithms for creating decision trees e.g. ID3, FT, J48, and C4.5 [45]. These algorithms generate decision trees by forming them top to down and identify the prime attribute that is to be chosen for classification of instances in a given dataset (i.e. it is in accordance with the selected target attribute). During construction of decision trees, 66.6% data is used for training the data whereas remaining 33.3% is used for testing the data. Table 2 represents an example of a database on which decision tree can be implemented.

Table 2 All Electronics customer database with examples representing when the customer buys computer
Full size table

Figure 1 represents its equivalent decision tree. This tree classifies the instances according to whether a customer buys the computer or not. Thus, the instance (age= youth, student = no, credit_rating = fair) can be sorted down left the most branch of the tree and can be classified as a negative instance (i.e. tree predicts buys_computer = no).

Fig. 1
figure 1

A decision tree for “buys_computer”, represents that a customer at AllElectronics is likely to purchase a computer

Full size image

Attribute selection

Attribute selection techniques can be categorized according to a number of criteria. One popular categorisation has coined the terms filter and wrapper to describe the nature of the metric used to evaluate the worth of attributes [47, 48]. Wrappers evaluate attributes by using accuracy estimates provided by the actual target learning algorithm. Filters (e.g. FilteredAttributeEval and FilteredSubsetEval), on the other hand, use general characteristics of the data to evaluate attributes and operate independently of any learning algorithm.

A proposed approach for qualitative usability feature selection with ranking (QUFSR)

In this section, a novel approach is proposed that is based upon the integration of usability testing [1], conventional heuristic evaluation [31,32,33], ISO 9241-151 guidelines [26, 27] and data-mining knowledge discovery process [12, 49]. The aim of this proposed approach is the identification of functional and problematic usability features from a huge number of websites by using data-mining techniques e.g. association rules and decision tree. So, an attempt has been made to prioritize problematic usability attributes belonging to websites by using attribute selection with ranker algorithm. Figure 2 represents the architecture of the proposed approach called, qualitative usability feature selection with ranking (QUFSRc) for interactive systems.

Fig. 2
figure 2

Proposed approach for qualitative usability feature selection with ranking (QUFSRc) for interactive systems. (*In this figure, ET stands for Evaluation Team)

Full size image

The proposed approach is decomposed into following five steps:

  1. I.

    Formulating QUFSR c process The QUFSRc approach commences with the formation of team-members for evaluation team. The team consists of a Ph.D. student having the research background in Human Computer Interaction (HCI), an associate professor in University and an HCI expert. The team decides to take a sample of websites, AWc = {AW1…AWn} that represents entire context C. The evaluation team considers a large number of websites for usability evaluation. This decision has been taken because the authenticity of any experiment’s results is mounted on large sample size to produce the results more representative of the whole population. In other words, the benefit of taking a large number of websites is to induce greater accuracy with proposed results [49, 50]. The ISO standards and required guidelines for usability evaluation are explored by team-members. Usability evaluation methods are reviewed and suitable methods are selected for evaluation of websites. Team also decides to use particular software that would be needed for implementation of data-mining knowledge discovery process.

  2. II.

    Participants The evaluation team prepares a list of participants that involved in the evaluation process of websites. Participants from varying age groups, different educational and professional backgrounds are considered in QUFSRc approach. An estimate of the minimum number of participants is also done by the team. Hundred participants are needed in the evaluation process of websites. Team also decides to include those participants that are frequent end-users of websites, so that comprehensive data collection process can be conducted.

  3. III.

    Formulation of heuristic questions This step begins with the formulation of guidelines that need to be evaluated by the participants. Most of ISO 9241-151 guidelines are considered that provides guidance on the human-centred design of software and web user interfaces with the aim of increasing usability [26, 27]. The evaluation team defines other heuristics-related questions addressing latest technological issues after evaluating the web systems as a whole. Each question belongs to a characteristic of academic websites that is to be evaluated by participants. Further, usability testing is conducted where end users would evaluate all these guidelines on the websites i.e. guidelines provided by ISO 9241-151 standard and heuristic guidelines defined by experts. The proposed approach employs both heuristic evaluation and usability testing methods so that comprehensive usability evaluation can be done.

  4. IV.

    Data collection for QUFSR c process The QUFSRc process is based upon qualitative usability data that is collected by involving the participants from different colleges within the university. For data collection, the Google form is created that collects the answers from all the participants. The Google form contains questions which are basically guidelines belonging to the characteristic of websites. Each question can be answered in “Yes”, “No” and “Sometimes”. If the participant finds the particular characteristic or guideline on the assigned website then he/ she is supposed to click on “Yes” as an answer. If the participant does not find a particular characteristic then he/ she clicks on “No”. If the participant recognizes particular characteristic on the website but that feature is not functional then he/ she opts for “Sometimes” as an answer. The collected data gets stored in CSV file format that is supported by Google form.

  5. V.

    Implementing data-mining knowledge discovery process within QUFSR c process Next step is to implement data-mining knowledge discovery Process on the collected data. The aim of this step is to discover hidden and unknown knowledge from stored data belonging to websites. The evaluation team decides to employ Weka workbench for the implementation of data-mining techniques. Three different techniques i.e. association rules, decision tree and filtered attribute evaluator with ranker algorithms are applied on stored data. Generally, association rules help in determining problems pattern in datasets. And decision trees assist in predicting the behavior of context of use of academic websites under evaluation. As an output, a list of association rules and decision trees are generated that should be analysed by the evaluation team. During analysis of output, evaluation team observes that both techniques help in classifying usability attributes into subsets of fully functional and problematic characteristics that belong to websites. Moreover, the QUFSRc process generates the output as a set of general usability problems patterns. Each detected usability problem pattern should be paraphrased as a usability problem in the standard format [1]. It also contains a subset of problematic usability attributes on which attribute selection algorithm is implemented to rank each problematic attributes. Basically, rank represents priority that is associated with each usability problems. High prioritized usability problem has the high effect of system or end users and should be corrected immediately. And developer should focus on proposing the solutions to these characteristic or usability problems so that usability of websites can be improved. In short, association rules, decision tree representation and attributes ranking technique collectively, can assist research practitioners and developers to focus more on the problematic characteristic.

Experimentation and results

In this section, obtained results are provided after applying proposed approach on top 50-academic websites. The first step is to select the evaluation team-members. The evaluation team is formed of 3-members: 1-Associate Professor in the university, 1-PHD student and an HCI expert. All members of the team have knowledge about usability and are frequent users of academic websites. The Internet Explorer, Mozilla Firefox, and Google Chrome browsers are used to visualize the academic websites. Ideally, the QUFSRc process must consider comprehensive usability evaluation of all academic websites in a context of use. But practically, it is not a feasible task because of huge associated resources in terms of time and cost. Therefore, a sample of academic websites AWc = {AW1…AWn} is taken to present entire context C. The guidelines have been followed to impart a representative sample for academic websites [49]. As a sample, websites of top 50-academic universities of India listed in National Institutional Ranking FrameworkFootnote 3 are considered. The evaluation team searches for usability guidelines from international standards for evaluation of academic websites. And then, decides to use guidelines from ISO 9241-151 [26, 27]. Some other heuristic questions belonging to academic websites are also defined by the evaluation team. The purpose of considering these heuristic questions is to incorporate latest technological trends related to academic websites. For example, in order to check whether academic websites are mobile optimized or not, a different category of heuristics highlighting the mobile related questions is considered. Similarly, the category for social media and security are defined. A Google form is created to receive the responses from all participants. It contains ISO 9241-151 and other heuristic guidelines as questions with three possible answers {Yes, No, Sometimes}. Google form helps in getting the answers within minimum time-limits. The output spreadsheet (with CSV format) is obtained and contains answers to all defined questions or guidelines related to the academic website. Moreover, the planning step considers the selection of particular software that is required for implementation of data-mining knowledge discovery process. So, Weka workbench is used for implementing 3-data-mining techniques e.g. association rule (PredictiveApriori algorithm), decision tree (ID3 algorithm), attribute selection (FilteredAttributeEvaluator with Ranker algorithm) etc. The aim of the proposed approach is to classify fully functional and problematic usability features. After which, the ranking of problematic usability features is to be done to find high prioritized usability problem so that immediate action can be taken and usability of websites can be improved. The proposed model is generic in nature but the definition of heuristics guidelines depends upon the type of web systems considered for usability evaluation. In other words, the proposed approach can be adapted to the broader range of web systems.

In the second step, participants from different educational and professional backgrounds (mainly from the Department of Computer Science and Information Technology) are listed for evaluating 50-academic website individually. All participants are frequent end users of academic websites. Hundred end users have participated in the evaluation process and hence results into hundred records in output spreadsheet. Each website is evaluated by two different users and chooses answers in {Yes, No, Sometimes}Footnote 4 for each defined question in Google form. Table 3 presents the details of end users that are considered for evaluation.

Table 3 End users details distribution
Full size table

For the experimental setup, an evaluation session is structured with a list of hundred participants from different colleges within universities. The strategy adopted for analyzing ISO 9241-151 and heuristic guidelines includes an overview of the websites under evaluation for about 5–10 min. In order to attract the participants in this research study, a motivational speech of 2–3 min has also been delivered.

As a result, participants have evaluated the websites with sincerity as they feel their responses or feedback can make a difference in design of academic websites. In Table 3 the age distribution of the users is ranging between 18 and 35 years. There are 13% married and 87% unmarried participants. There are 77 male and 23 females participants. 71% of the users are invited to conduct the experiment within a laboratory meanwhile the rest of participants are being asked to conduct experiment remotely from their workplace through emailing them a link to Google form, URL of assigned website and description of steps to execute the entire procedure. As far as the educational background of the participants is concerned, 67% are B.Tech, 9% M.Tech Students, 9% BCA, 4% MCA, 4% MBA and 7% are Ph.D. Students. 13% are college and university assistant professors, 71% are students, 3% are govt employees meanwhile 13% are corporate professionals who have finished their master level degrees.

In the third step, the evaluation team has chosen usability testing [1] and heuristic evaluation [31,32,33] as both are frequently used methods for usability evaluation. The team considers 93-general usability guidelines collectively from ISO 9241-151 and other heuristics. Table 7 represents these guidelines belonging to 16-different categories.Footnote 5 Each guideline belongs to the various characteristic of academic websites that are to be evaluated.

The fourth step commences with the evaluation of websites by the participants. Data collection process is executed by creating a Google form. Each guideline is represented as a question in the form. Range values are also defined for each possible answer in the form. Figure 3 represents a partial view of collected data belonging to the content category of usability data. Each question that user evaluates, is associated with an attribute name.

Fig. 3
figure 3

Partial outlook of usability attributes belonging to content category in .csv file format

Full size image

In Fig. 3 of output spreadsheet, these attribute names are represented as column headings and each row corresponds to complete usability evaluation of a specific academic website. Hence, top fifty academic websites are evaluated and total ninety three usability guidelines are defined and their answers are stored into output spreadsheet, resulting into hundred records (each website is evaluated by two different users). In this way, answers of all defined questions in output spreadsheet represent usability data.

After collecting and storing qualitative data, the last task in the QUFSRc process is to implement data-mining knowledge discovery process. The aim of this step is to discover hidden and unknown knowledge from stored data belonging to academic websites. Evaluation team chooses association rules, classification and attribute selection component from Weka workbench. The team then implements PredictiveApriori, ID3, and FilteredAttributeEval (InfoGainAttributeEval) with Ranker algorithm on obtained usability data. Generally, association rules are applied to find problem patterns in datasets and the decision trees help in predicting the behavior of context of use under evaluation. Attribute selection (i.e. InfoGainAttributeEval) is used to evaluate the worth of an attribute by measuring the information gain with respect to the class. Whereas, ranker algorithm ranks attributes by their individual evaluations. In next subsections, these three techniques are implemented and their respective outputs are analysed by the evaluation team.

Generally, association rules are applied to find problem patterns in datasets. In this study, PredictiveApriori algorithm from WEKA workbench is selected and implemented on usability data. As an output, PredictiveApriori algorithm has generated hundred association rules. So, those association rules that explain equivalents usability problems are jointly considered by the evaluation team. Table 4 represents a comprehensive list of association rules belonging to the content category of usability data.

Table 4 Obtained list L1 of association rules belonging to content category after applying PredictiveApriori in WEKA
Full size table

In the obtained list, association rules #1 to #8 and #15, point to problematic usability attributes that relates to the absence of a Scope_of_Multi_Language attribute. These rules reflect that participants do not find multi-language support on different web pages of academic websites. Also, the usability problematic attributes in rule #1 to #8 and #15 exhibit significant relationships among themselves. The evaluation team analyses this output list to create two subsets of usability attributes. One subset represents optimal usability attributes that are fully-functional on academic website whereas another subset represents problematic usability attributes.

The first subset contains following attributes i.e. Date_of_Last_Update_Available, Communication_with_Site_Owner, Online_User_Feedback, Multi_Language, and Scope_of_Multi_Language. These attributes are considered as problematic attributes as their frequency for “no and sometimes” is high for their respective values. Another subset contains Content_Up_To_Date, News_Comprehensibility, and Contact_Data_Available and is identified as optimized attributes as their frequency for “yes” is high for their respective values. Developers and usability professionals must focus on identified problematic attributes as they are related to significant ISO-guidelines. Developers must emphasize on these guidelines carefully during designing and develop novel websites. In other words, these problematic attributes point to critical interface elements of academic websites that should be removed immediately to achieve high usability of existing and novel websites.

Moreover, evaluation team decides to keep these identified problematic attributes as “target” and acquires decision trees that assist in predicting the behavior of context of use of academic websites under evaluation. In this study, the ID3 algorithm from WEKA workbench is selected and implemented on usability data. In WEKA, 66.66% of available usability data is considered as training set and remaining data is used as test set. For admissible decision trees, the threshold values are set to 96% of correctly classified instances. Table 5 represents obtained decision tree after applying the ID3 algorithm on the content category of usability data.

Table 5 Obtained decision tree after applying the ID3 algorithm on the content category of usability data (target attribute Multi_Language with possible values {yes, no, sometimes}) (visualization provided by WEKA platform)
Full size table

In this resulting tree, “target” attribute belongs to the Multi_Language attribute of content category. In the same way, various decision trees are attained for different categories of usability data. Each line in the text representation of decision tree indicates the value assigned to inner node i.e. other attributes of the content category. And it is presented as “Attribute_Name=value”. Each “|” indicates to next level in the decision tree. Text representation also contains “: class_value” that indicates the value which is assigned to target attribute. Each branch of the decision tree is read as “if-then” rule. For example, the first 4-lines of Table 5, can be read as: if (Scope_of_Multi_Language=no) and (Date_of_Last_Update_Available=no) and (Online_User_Feedback=no) and (Communication_with_Site_Owner=no) then (Multi_Language=no).

In Table 5, it is important to observe that values stored in the attributes Scope_of_Multi_Language, Date_of_Last_Update_Available, Online_User_Feedback, and Communication_with_Site_Owner are significant to determine the value of the Multi_Language attribute. This is due to fact that these attributes are near to root of the obtained decision tree. And these attributes validate the subset of problematic attributes (As obtained by applying association rules i.e. PredictiveApriori algorithm). On the other hand, the values stored in News_Comprehensibility, Content_Up_To_DateandContact_Data_Available attributes are not particularly relevant for predicting the value of the Multi_Language attribute. By just looking at output spreadsheet, the evaluation team cannot easily detect these problem patterns belonging to academic websites. This stored information is previously unknown and hidden for evaluation team. Besides, the team considers this information valuable as it is not based on any intuition or on simple observation. This text-based representation of decision tree can be considered as new knowledge about usability problematic attributes. Each attribute represents characteristic or features of academic websites. Evaluation team infers that these characteristic are not fully functional on academic websites. These characteristics relate with significant guidelines (i.e. ISO 9241-151 guidelines as mentioned in Table 7) that should be carefully considered during usability evaluation. Developers need to focus on these guidelines to remove existing errors from academic websites. Identification of these problematic usability attributes can also assist developers in evaluating new websites belonging to the same context of use.

QUFSRc process paraphrase identified usability problem patterns that are reported and documented in a standard format [1]. An example related to the content category is shown in Table 6. The frequency represents the number of association rules that validate the pattern with confidence and support values of the rule. The evaluation team also provides certain comment and recommendation that belong to identified problematic pattern.

Table 6 Paraphrased identified usability problem
Full size table

Another subset that includes Content_Up_To_Date, News_Comprehensibility and Contact_Data_Available are identified as fully functional attributes under the content category. It is important to note that obtained subsets impart initial guidelines to identify fully functional and problematic attributes under each defined categories. After recognizing subset of problematic usability attributes, evaluation team decides to prioritize each problematic attribute. Attribute selection i.e. InfoGainAttributeEval with Ranker algorithm is implemented on the content category of usability data. Figure 4 represents the ranking of problematic attributes belonging to content category. The attribute Scope_of_Multi_Language is ranked first in the list which represents that it is the high prioritized problem. The Multi_Language attribute is second high prioritized usability problem that is followed by the next usability problematic attribute. The high prioritized problem is more frequent and has the higher effect on systems or end users. The prioritization of problem is also related to severity assessments. The severity associated with usability problem is defined as an assessment of quantum of inconvenience the end users would face during evaluation of web systems. Another benefit of prioritizing the problem can be considered as an instrument for the utilization of design resources. It can also be considered as a device that provides guidance to designers, developers and usability professionals about the order in which usability problems must be addressed. Developers need to focus on this prioritized problematic attributes so that errors can be removed from existing academic websites. By doing so, developers and usability professionals can improve the usability of academic websites. If appropriate, the use of prioritizing usability issues is to begin a discussion for potential solutions to identified prioritized problems.

Fig. 4
figure 4

Ranked problematic attributes belonging to content category after applying FilteredAttributeEval(InfoGainAttributeEval) and Ranker algorithm on usability data

Full size image

Figure 5 provides graphical representation in which radar charts are used to represent the problematic usability attributes with their associated rank for content category belonging to top fifty academic websites. Rank represents the priority associated with each problematic attribute. Attribute with a lesser value of rank should be addressed immediately as participants do not find that particular attribute functional at all. Again, this highlighted information is previously unknown and hidden for evaluation team and considered to be new knowledge that helps in identifying, classifying and ranking the usability problems concerning the context of use. Similarly, for other categories, problematic usability attributes are identified and ranked by using the proposed approach. No issues are identified in two-categories i.e. {Headings_Titles_Labels and The Home_Page}. Participants find fully functional guidelines in these two categories. In Scrolling and Paging category, horizontal scrolling is identified as a major issue by most of the participants. Another issue is addressed in Internationalization category and participants do not identify supported languages on academic websites. Also, participants do not find academic websites to be highly secured under Security category. Most of the issues are also found in Search and Social Media categories. Further, out of ninety three guidelines, evaluation team identifies that forty seven guidelines are fully functional guidelines. Participants easily locate 50.53% guidelines on websites and identify them as fully functional usability features. Whereas, remaining forty six guidelines are related with problematic attributes. These problematic attributes belong to usability features that are either missing or participants cannot locate them easily on the academic websites. Due to these reasons, 49.46% of characteristics are considered as problematic usability features that are not functional on the academic websites. Research practitioners, usability experts, and developers must focus on these problematic features (or attributes) to remove errors from websites and to improve the usability of academic websites.

Fig. 5
figure 5

Ranked problematic usability attributes belonging to content category

Full size image

Conclusions and future work

The main challenge in usability engineering is the detection of common usability problems in any system in a given context of use as a whole. Proposing suitable solutions for these problems can become valuable in usability engineering or for software development in various aspects. On the contrary, usability professionals can depend upon these identified usability problems in a given context when novel interfaces of systems in similar context are evaluated. Moreover, this contextual information regarding usability problems can assist software developers to restrain from these potential usability problems during the development of any novel systems under the same context.

Qualitative usability testing and heuristic evaluation methods impart significant procedure for evaluating the usability of any system. Although, these methods become restricted during collective analysis of a huge number of systems related to the context of use. In order to overcome these existing problems, QUFSRC approach is proposed. The proposed approach represents a novel dimension of qualitative usability testing and heuristic evaluation that is based upon the integration of conventional usability engineering methods and data-mining knowledge discovery process.

The advantage of QUFSRC approach is that data-mining knowledge discovery algorithms can be executed only once to attain desired results. The execution is performed on pre-decided criterion i.e. target attribute in regard to decision tree. Robust automatization in data-mining knowledge discovery process within QUFSRc approach is also a beneficial trait [5]. It not only improves the systematization but also improves predictability in the results of usability evaluation. It also minimizes the prejudices by evaluation team during consideration of a huge amount of qualitative data. It is significant to note that evaluation team is constantly responsible for controlling the QUFSRc approach. This approach has improved the decision-making skills of the team, but it has not changed the ultimate discussion within evaluation team regarding the results that are obtained after applying usability testing and heuristic evaluation methods.

From the experiments conducted in this study, we conclude that by the integration of data-mining knowledge discovery process for the qualitative usability evaluation for a context of use as a whole can be executed successfully as the case for real world problems (i.e. academic website). The significant advantage of the proposed approach is that the several intuitions that are expressed informally during qualitative usability testing and heuristic evaluation can be evaluated appropriately through data-mining knowledge discovery techniques. On the contrary, evaluation team uncovers the hidden or unknown relationships among usability data. The significant application of the proposed approach is to identify the qualitative usability problems of any prototype related to a specific context of use and to prioritize these identified usability problems. Problem prioritization can be considered as an integrated element of design and evaluation. It further, intends to affect the designer’s allocation of their efforts and time. Fixing the high prioritized usability problem would significantly overshadow the insights attained from it, as more severe usability problems engross substantial user’s resources and evaluation time.

The proposed approach also shows that there is the need for an improvement in academic websites as their structuring is not completely following the guidelines provided by ISO 9241-151 standard [26, 27]. So it triggers substantial amount of the changes/corrections in existing academic websites as per standards. As a result of which a large number of potential end users can be retained with high satisfaction and acceptance.

In brief, the integration of data-mining knowledge discovery techniques into conventional qualitative usability testing and heuristic evaluation methods enables that these methods can go beyond than testing of systems, and hence provides the potential qualitative usability evaluation of a context of use.

As compared to prior related works, this paper highlights certain differences as follows: (a) To the extent of our knowledge, there is no other alternative that encompasses conventional qualitative usability testing, heuristic evaluation and ISO 9241-151 guidelines with data-mining knowledge discovery, as explained in this paper. (b) Usability evaluation of academic websites by using qualitative usability testing and heuristic evaluation methods are strong enough to include real user insights and produce more extensive results. (c) ISO 9241-151 guidelines and other latest technological issues related to academic websites (e.g. mobile, social media and security etc.) are considered for evaluation of academic websites. Other studies [8] have ignored these major technological key points(In this study, sixteen different categories are defined for evaluating ninety three usability guidelines related to academic websites) (d) Usability attribute selection method with the ranking algorithm is not adopted in any of the usability studies to prioritize problematic attributes belonging to academic websites. An attempt has been made to highlight the priority of existing usability problem. So that developers and usability professionals can remove high prioritized problems from the academic websites and can improve the usability of websites.

There is a lack of usability studies that focus on severity assessment to prioritise the usability problems and its impact on the outcome of prioritization problem. Further, a recommendation can be made for future studies to incorporate severity assessment as part of the method. The future work can also be pursued in implementing advanced algorithms for association rule, decision tree, and attribute selection. Such algorithms can impart more comprehensive and exhaustive results. Besides, the usability data that is stored into a database can be applied further for statistical analysis. By performing statistical analysis, we can perform the quantitative assessment of usability for websites. And the ranking of academic websites of top fifty universities can be done.

Notes

  1. QUFSRc represents qualitative usability feature selection with ranking for context of use.

  2. Top-50 academic websites listed in National Institutional Ranking Framework are considered.

  3. https://www.nirfindia.org/univ.

  4. Yes-{if guideline is found on assigned website and it is functional in nature}, No-{if guideline is not identified on assigned website}, Sometimes-{if guideline is found on website but it is not functional in nature}.

  5. 16-different categories are as follows: Content Organization, Design Process, Mobile, Navigation, Search, Security, Social Media, Links, {Headings, Titles, and Labels}, {Scrolling and Paging}, Page Layout, The Home page, Accessibility, Optimizing the User Experience, {Graphics, Images, and Multimedia}, and Internationalization.

References

  1. Dumas SJ, Redish CJ (2000) A practical guide to usability testing. Intl. Specialized Book Service Inc., Portland

    Google Scholar 

  2. Institute of Electrical and Electronics Engineers (1990) IEEE standard glossary of software engineering terminology IEEE std. 610.12-1990. Institute of Electrical and Electronics Engineers, Los Alamitos

    Google Scholar 

  3. International Organization for Standardization, ISO 9241-11 (1998) Ergonomic requirements for office work with visual display terminals (VDTs). Part 11: guidance on usability. International Organization for Standardization, Geneva

    Google Scholar 

  4. Maguire M (2001) Context of use within usability activities. Int J Hum Comput Stud 55:453–483

    Article  MATH  Google Scholar 

  5. Ivory YM, Hearst AM (2001) The state of the art in automating usability evaluation of user interfaces. ACM Comput Surv 33:470–516

    Article  Google Scholar 

  6. Nielsen J (1993) Usability engineering. M. Kaufman Publishers Inc, San Francisco

    MATH  Google Scholar 

  7. Gonzalez MP, Granollers T, Lores J (2006) A hybrid approach for modelling early prototype evaluation under user-centred design through association rules. In: Proceeding XIII Intl. Workshop Design. Specification and verification of interactive system DSV-IS ’06. Sponsored by ACM SIGCHI. Eurographics, IFIP WG 13.5. Dublin. Ireland (Lecture Notes in Computer Science Springer Series)

  8. Gonzalez MP, Lores J, Granollers (2008) Enhancing usability testing through datamining techniques: a novel approach to detecting usability problem patterns for a context of use. Inform Softw Technol 50:547–568

    Article  Google Scholar 

  9. Tiedtke T, Martin C, Gerth N (2002) Awusa. A tool for automated website usability analysis. In: 9th International Workshop DSVIS

  10. Alipio J, Pocas J, Azevedo P (2002) Recommendation with association rules: a web mining application. In: Data mining and warehouses conference IS-2002

  11. Finlay J (1997) Machine learning: a tool to support improved usability. Appl Arti Intell 11:633–665

    Article  Google Scholar 

  12. Sagar K, Saha A (2016) Enhancing usability inspection through data-mining techniques: an automated approach for detecting usability problem patterns of academic websites. In: International conference on intelligent human computer interaction. Springer, Cham, pp 229–247

  13. Moumane K, Idri A, Abran A (2016) Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards. SpringerPlus 5:1–15

    Article  Google Scholar 

  14. Jankowski J, Grabowski A (2015) Usability evaluation of vr interface for mobile robot teleoperation. Int J Hum Comput Interaction 31:882–889

    Article  Google Scholar 

  15. Ji GY, Park HJ, Lee C (2006) A usability checklist for the usability evaluation of mobile phone user interface. Int J Hum Comput Interaction 3(3):207–231

    Article  Google Scholar 

  16. Kjeldskov J, Stage J (2004) New techniques for usability evaluation of mobile systems. Int J Hum Comput Stud 60:599–620

    Article  Google Scholar 

  17. Erickson W, Trerise S, Lee C (2013) The accessibility and usability of college websites: is your website presenting barriers to potential students? Commun College J Res Pract 37:864–876

    Article  Google Scholar 

  18. Fernandez A, Insfran E, Abrahão S (2011) Usability evaluation methods for the web: a systematic mapping study. Inform Softw Technol 53:789–817

    Article  Google Scholar 

  19. Aziz MA, Isa WARWM, Nordin N (2010) Accessing the accessibility and usability of malaysia higher education website. In: IEEE in international conference on user science and engineering. pp 203–208

  20. Panach IJ, Juristo N, Valverde F (2015) A framework to identify primitives that represent usability within model-driven development methods. Inform Softw Technol Elsevier 58:338–354

    Article  Google Scholar 

  21. Lee S, Cho JE (2007) Usability evaluation of korean e-government portal. In: Universal access in human-computer interaction applications and services. Springer, New York, pp 64–72

  22. Sauro J, Kindlund E (2005) A method to standardize usability metrics into a single score. ACM HCI, New York

    Book  Google Scholar 

  23. Bak JO, Nguyen K, Risgaard P, Stage J (2008) Obstacles to usability evaluation in practice: a survey of software development organizations. Proc. NordiCHI. ACM Press, New York

    Book  Google Scholar 

  24. Ardito C, Buono P, Caivano D, Costabile MF, Lanzilotti R, Bruun A, Stage J (2011) Usability evaluation: a survey of software development organizations. In: Proc. SEKE, Knowledge Systems Institute Graduate School, Skokie

  25. Bruun A, Stage J (2012) Training software development practitioners in usability testing: an assessment acceptance and prioritization. OZCHI’12, November 26–30

  26. Bevan N (2005) Guidelines and standards for web usability. In: Proceedings of HCI International, vol 2005. Lawrence Erlbaum, New Jersey

  27. http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=37031. Accessed 5 Aug 2016

  28. Teruel AM, Navarro E, Jaquero LV et al (2014) A cscw requirements engineering case tool: development and usability evaluation. Inform Softw Technol 56:922–949

    Article  Google Scholar 

  29. Rodriguez DF, Acuna TS, Juristo N (2015) Design and programming patterns for implementing usability functionalities in web applications. J Syst Softw 105:107–124

    Article  Google Scholar 

  30. McClellan AM, Karumur PR, Vogel IR et al (2016) Designing an educational website to improve quality of supportive oncology care for women with ovarian cancer: an expert usability review and analysis. Int J Hum Comput Interaction 32:297–307

    Article  Google Scholar 

  31. Somervell J, McCrickard D (2004) Comparing generic vs. specific heuristics: Illustrating a new UEM comparison technique. In: Proceedings of the human factors and ergonomics society. pp 2480–2484

  32. Hvannberg ET, Law EL, Lárusdóttir MK (2007) Heuristic evaluation: comparing ways of finding and reporting usability problems. Interact Comput 19(2):225–240

    Article  Google Scholar 

  33. Nielsen J (1994) Heuristic evaluation. In: Nielsen J, Mack RL (eds) Usability inspection methods. Wiley, New York

    Chapter  Google Scholar 

  34. Dubey KS, Gulati A, Rana A (2012) Usability evaluation of software systems using fuzzy multi-criteria approach. Int J Comput Sci Issues 9(2):1694

    Google Scholar 

  35. Dubey KS, Rana A, Sharma A (2012) Usability evaluation of object oriented software system using fuzzy logic approach. Int J Comput Appl 43(19):0975–8887

    Google Scholar 

  36. Gupta D, Ahlawat AK (2017) Taxonomy of GUM and usability prediction using GUM multistage fuzzy expert system. Int Arab J Inform Technol 16(3):2019

    Google Scholar 

  37. Gupta D, Ahlawat AK, Sagar K (2017) Usability prediction & ranking of SDLC models using fuzzy hierarchical usability model. Open Eng 7(1):161–168

    Article  Google Scholar 

  38. Gupta D, Ahlawat AK (2016) Usability determination using multistage fuzzy system. Procedia Comput Sci 78:263–270

    Article  Google Scholar 

  39. Mikut R, Reischl M (2011) Data mining tools. In: Wiley interdisciplinary reviews: data mining and knowledge discovery, vol 1. pp 431–443

  40. https://rapidminer.com/. Accessed 23 Sept 2016

  41. http://www.rdatamining.com/. Accessed 23 Sept 2016

  42. Demsar J, Zupan B, Leban G (2004) Orange: from experimental machine learning to interactive data mining white paper. Faculty of Computer and Information Science University of Ljubljana, Ljubljana

    Google Scholar 

  43. Ian H, Witten Frank E (2005) Data Mining: Practical machine learning tools and techniques. M. Kaufman Publishers Inc., San Francisco

    MATH  Google Scholar 

  44. WEKA. http://www.cs.waikato.ac.nz/ml/weka/. Accessed 14 Oct 2016

  45. Han J, Kamber M (2000) Data mining: concepts and techniques. M. Kaufmann, San Francisco

    MATH  Google Scholar 

  46. Mitchell T (1997) Machine learning. McGraw Hill, New York

    MATH  Google Scholar 

  47. Hall Mark A, Holmes G (2003) Benchmarking attribute selection techniques for discrete class data mining. IEEE Trans Knowl Data Eng 15(6):1437–1447

    Article  Google Scholar 

  48. Kohavi R, John HG (1997) Wrappers for feature subset selection. Artif Intell 97(1):273–324

    Article  MATH  Google Scholar 

  49. Pyle D, Kamber M (1999) Data preparation for data mining. M. Kaufmann, San Francisco

    Google Scholar 

  50. http://classroom.synonym.com/advantage-doing-experiments-large-samples-13175.html. Accessed 5 Jun 2017

Download references

Authors’ contributions

KS evaluated top-50 academic websites using ISO 9241-151 guidelines, collected the data, implemented three data-mining techniques, detected usability problematic patterns, analysed and interpreted the results. AS has aided in the analysis and presentation of the results mentioned in this study. Both authors read and approved the final manuscript.

Acknowledgements

Deepak Gupta and Amit Kumar Tiwari are gratefully acknowledged for their assistance in providing their suggestions for improvement of this paper. We also thank Associate Professors of Guru Gobind Singh Indraprastha University for their help in conceptualizing this work.

Competing interests

The author declares that they have no competing interests.

Availability of data and materials

We don’t wish to share our data.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Funding

Not applicable.

Statement

We state that this research was carried out as per our institution’s guidelines.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations

  1. University School of Information and Communication Technology, GGSIPU, sector 16-c, Dwarka, Delhi, 110078, India

    Kalpna Sagar & Anju Saha

Authors
  1. Kalpna Sagar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anju Saha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kalpna Sagar.

Appendix

Appendix

See Table 7.

Table 7 Usability evaluation guidelines for academic websites
Full size table

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sagar, K., Saha, A. Qualitative usability feature selection with ranking: a novel approach for ranking the identified usability problematic attributes for academic websites using data-mining techniques. Hum. Cent. Comput. Inf. Sci. 7, 29 (2017). https://doi.org/10.1186/s13673-017-0111-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13673-017-0111-8

Keywords