Table 1 Issues to have an unbiased rating scale (summarized from [17, 19, 20,21,22])
From: Rethinking self-reported measure in subjective evaluation of assistive technology
Issues | Explanation |
|---|---|
(1) The connotations of category labels | Rating descriptor words require some more thought. Not equal-interval scale may cause biased scale. Example: terrible__horrible__awful __fair __slightly good__all right__reasonably good |
(2) Effect of response alternatives on interpretation of the question | The response alternatives can affect the interpretation of the question. Knowledge of this phenomenon makes it easy to influence the responses of subjects. Example: “how often have you considered quitting your job?” |
(3) Implicit assumptions of the question | Some questions are biased because of an implicit assumption made by the question. Example: intrinsic or germane cognitive load? |
(4) Forcing a choice | A forced-choice rating scale will bias results by eliminating the undecided and/or those with no opinion. In disability study, this is a crucial consideration |
(5) Unbalanced rating scales | Generally, rating scales should be balanced, with an equal number of favorable and unfavorable response choices. Example: (unbalanced) “Excellent,” “very good,” “good,” “fair,” “poor.” This scale is unbalanced, with three favorable and only one unfavorable response choice |
(6) Order effects in rating scales | Traditionally, researchers present the most positive items in the scale first (e.g., “strongly agree,” “extremely interesting,” or “extremely satisfied”) and the most negative items last (“strongly disagree,” “very boring,” or “extremely dissatisfied”) |
(7) The direction of comparison | Many surveys contain questions of comparison, where respondents are asked to compare two stimuli |
(8) The number of points | Ideally, a rating scale should be consistent enough points to extract the necessary information. Variability can be improved by using scales with too many points |
(9) Context effects | Many surveys consist of a series of questions whose purpose is to help the researcher determine which factors correlate most strongly with the subjects’ overall opinion. Some questions may influence by subsequent questions |