MobileActive.org

Many mobile and tech for development projects elicit enthusiastic responses by the target constituency when they are asked, but then go on to not be used. "Yours is Better!" Participant Bias in HCI investigates the influence of researchers and developers on how beneficiaries react to new services and products, specifically the social and behavioral reasons why users may claim to like a project or find it useful. In an effort to understand the relationship between interviewers and interviewees, two researchers ran an experiment to test the effects of participant bias. Covering interviewers with 450 residents of Bangalore, India, the experiment tested three hypotheses:

• H.1 If participants believe that the interviewer favors a technology, their responses will be biased to favor it as well.
• H.2 If the interviewer is a foreign researcher requiring a translator, participants’ responses will be even more biased towards the technology favored by the interviewer.
• H.3 Participants will express a preference for an obviously inferior technology if they believe it is favored by the interviewer.  (quoted from "Yours is Better" Participant Bias in HCI)

To test these, the researchers ran two experiments asking both university students and rickshaw drivers to rank the video quality of two smartphone video players. The first experiment found that when the interviewers associated themselves with a particular player (saying "This phone uses my new player"), participants were more likely to choose it, despite the fact that the videos were identical. The second experiement found that associating the interviewer with a player increased the number of participants saying they prefered it, even if the associated video was of a lower quality than the other option.

Nicola Dell, one of the primary researchers on the project explained, "I was personally overwhelmed by the amount of bias that we saw; initially going into it I thought, 'Well, we might see some,' but I didn't have any idea of the magnitude."

The Experiment

Over the course of five weeks, separate groups of fifty participants were asked to compare two video clips on identical Windows smartphones. The participants were either from a local university or rickshaw drivers on a main road, and were all male; the interviews were run by either a foreign, female, English-speaking interviewer or a local, female, Kannada-speaking interviewer.

In the first experiment, each interviewer indicated that they had developed one of the video players (even though they were actually identical) and asked participants to rank which video player was better, or if they were the same, to test if associating themselves to the project would result in a skewed preference for "their" work.

In the second experiment, the researchers showed an obviously low quality clip (with reduced resolution and frame rate) and a high-quality clip. The interviewers asked the first group of 50 students and 50 rickshaw drivers which video they liked better without associating themselves to the project, then repeated the experiment on new groups claiming they had developed the player that was showing the lower quality video.

The Results

In the first experiment, in which the researchers showed two identical clips, when the foreign interview asked the rickshaw drivers to rate the videos, 35 drivers chose the video associated with the interviewer, seven choose the other video, and eight said the videos were the same. When the local interviewer asked the rickshaw drivers their preferences, 19 chose the video associated with the interviewer, and nine chose the other video, and 22 said the videos were the same.

When the foreign interviewer asked 50 university students which of the identical video clips they preferred, 25 choose the video associated with the interviewer, 14 thought they were the same, and 11 chose the other video. When the local interviewer asked the university students which video they preference, 24 preferred the interviewer's video, 12 thought they were the same, and 14 chose the other video.

In the first experiment, associating themselves to a particular video resulted in the majority of the participants choosing the interviewers' video as superior, even though there was no difference in quality between the two. The bias was especially pronounced when the foreign researcher was in charge of the interview.

In the second experiment, the interviewers first showed the obviously degraded clip without associating themselves to either player. Both the university students (47 out of 50) and the rickshaw drivers (38 out of 50) overwhelmingly chose the high-quality clip as the better video player. But when the interviewers associated themselves with the low-quality video (again saying "This phone uses my new player"), the results changed.

When the foreign interviewer asked the university students which video they preferred, 38 chose the high-quality video while ten said they preferred the low-quality video and two participants said they were the same. When the foreign interviewer asked the rickshaw drivers which video they preferred, the majority said they preferred the low-quality video (27 preferred the low-quality, 19 preferred the high-quality, and 4 said they looked the same). However, when the local interviewer asked the rickshaw drivers which video they preferred, the majority still preferred the high-quality video, albeit at a lower rate than with the unassociated video (29 preferred the high-quality video, 18 preferred the low-quality video, and 3 said the were the same).

In this case, the responses from the rickshaw drivers were dependent on who was doing the interview – with the foreign interviewer, the majority of the drivers preferred the lower-quality, associated video, which with the local interviewer, they still preferred the higher-quality video.

Below are two charts from the report illustrating the results:

Results from testing two identical videos:

Results from testing two videos of different quality:

Conclusion

The experiment highlights the weaknesses in subjective evaluation, namely that social and behavioral factors can influence responses, leading to biased results. The report recommends focusing on objective evaluation tactics in order to better understand how beneficiaries view the project and to remove some of the effects of interviewer influence. Dell says that understanding bias is necessary to honestly and openly evaluating one's own work; she explains, "The first and most important thing is just to be aware of it; be aware that people might just be telling you what you want to hear or what they think they should say, and that it's not necessarily the truth." Recognizing response bias and working to mitigate it by not asking leading questions or pressuring respondents is key for practitioners who want to evaluate the effectiveness of their tech projects.

"Yours is Better" Participant Bias in HCI will be presented formally at CHI2012, you can read a copy of the paper here