The taste test and confounding variables
The taste test
Confounding variables
This post was published in Psychological Review
Photo by Photo by Battlecreek Coffee Roasters
The taste test: cola and confounding variables
You’ve been invited to take part in a taste test. You’re given two identical looking, but different, drinks to taste. You’re not told anything about them, other than that you should express a preference for one or the other. In the one cup — labelled ‘L’ — is a sugary, sickly cola drink. In the other cup — labelled ‘S’ — is a an identical looking, but different sugary, sickly cola drink. You don’t like either, really. But you indicate that, of the two, you prefer ‘S’.
A thought
You stop and think for a second. Your knowledge of psychology and experimental design alerts you to a potential design flaw. The drink labelled ‘S’ was presented to you on your right. And you drank it with your preferred hand. Maybe, just maybe, you weren’t expressing a preference for the taste of the drink, so much as a preference for drinking with your right hand?
You put this possibility to the person behind the desk who seems to be running the show.
“It’s okay,” they reassure you. “Half of the participants in this trial will have the drinks presented the other way round. So, when the researchers are analysing the data, they will be able to see whether there is an overall preference, averaged across everyone, irrespective of which one was on the right, and which on the left.”
You walk away from the table, slightly disheartened that you hadn’t been able to destroy the legitimacy of the taste test with one devastatingly incisive question but reassured that you have taken part in something that will have some scientific credibility.
A second thought
Another possibility crosses your mind. You tasted the ‘L’ drink first, and then the ‘S’ one. Maybe that influenced your response? You put your mask back on (there was a pandemic at the time) and do an about-turn. Could it be that the second sip of a sickly-sweet cola is less unpleasant than the first? Have your taste buds simply got a little bit used to the taste, second time round?
The person behind the table greets you with a ‘you again?’ look. You inarticulately explain your latest sceptical take on the design of the taste test. They patiently explain that half the participants will get the ‘S’ drink first, and half will get the ‘L’ drink first. So, again, the sequence in which each individual tasted the drinks cannot bias the outcome of the results overall. You recognise this as the principle of ‘counterbalancing against order effects’ in action and retreat once more.
You're on a roll, thoughts-wise
“Oh! Hang on!” you expostulate, silently. “Wait one damn minute!” As you re-approach the table, you undertake an internal risk assessment and pre-empt the ‘look’ with a cheerful, “Hi there. It’s me again!”
“Welcome back, ‘me again’,” comes the polite response. The words 'teeth' and 'gritted' come to mind.
You carry on regardless. “So, I was thinking. You know all that psychological stuff about ‘expectations’ and ‘self-fulfilling prophecies’?”
The person behind the taste test desk nods wearily. You get the impression that they are well aware of Rosenthal and Fode's (1963) classic work in this area.
“Well, let’s just say that the researchers actually expect me to choose a specific drink — ”
“They don't.”
“— and that furthermore you communicated that expectation to me — ”
“I didn’t.”
“— and so it's in fact you who influenced me to make the choice that I made. You might think you didn’t, but have you ever heard of ‘Clever Hans’?” You introspect for a second, and reassure yourself that you can recall the original reference to Pfungst's (1911) famous work.
“We’re testing two different colas. And I don’t know which cola is in which glass. So, I can’t convey any ‘expectation’ to you. I simply don’t know which is which.” This person has the patience of a saint.
“You don’t know which is which? What’s the point then?”
“Well, someone, somewhere knows which is which, obvs, otherwise this whole thing would be a complete waste of time. But not me. I don’t know which drink is in which glass so I cannot influence you in any meaningful way. It’s called a ‘double blind’ design. That’s the whole reason for the letter labels. One of the drinks that we’re testing is always ‘S’, whichever side it’s presented, and in whatever sequence, and the other one is always ‘L’. When the data are analysed, the researchers know which drink is which. But I don’t. That way I cannot bias the results. Are we done?” Even saints run out of patience eventually.
“They really have got all the bases covered, haven’t they,” you find yourself thinking as you mask up once more and head home on the bus. It’s not until later that evening, when you are reading about ‘experimenter effects’ in your favourite psychology textbook (Banyard & Grayson, 2008), that one final thought occurs to the sceptic in you. Excitedly, you start a library search.
It turns out that…
You stop dead in your tracks. On the screen in front of you is the abstract from a study by Woolfolk, Castellan and Brooks (1983).
Sixty subjects tasted cola from two cups, one marked L, the other marked S. The same product (either Pepsi or Coke) was placed in both cups. Whether both cups contained Pepsi or Coke, subjects overwhelmingly reported cup S contained the better-tasting product… It was concluded that a taste comparison of colas should avoid using any labels, even presumably neutral ones like letters of the alphabet, since such labels may have more powerful influences on product comparisons than taste differences.
The implications flood through you like an injection of evergivingcoolfizzing energy and you fire off an email to your tutor, telling them about your discoveries. Sure, it's midnight, but they won't mind because email is asynchronous. Later the next morning you find this reply.
Email from your tutor
'Well spotted! Coincidentally I have often used this very example to teach about experimental design. And it just goes to show how difficult it is to control against potentially confounding variables. Who would have thought that there would be a general preference for the letter 'S' over the letter 'L'? As it happens, I've been asking our psychology undergraduate students about which letter they prefer (before telling them about the effect, of course) for nearly ten years. The data I've collected are shown in Figure 1. Needless to say, my results are based on a haphazard opportunity sample of participants (varying from 100 to 250 per year), so be cautious in interpreting them. However, you will see that the effect is very stable! Across all these cohorts over 70 per cent of those asked express a preference for the letter 'S'.
Figure 1: Undergraduate psychology student preferences for the letters 'L' and 'S'
'There are a number of theories as to why this might be the case. One has to do with the frequency with which these letters appear in our own given names (the so-called 'name-letter effect'). However, it doesn't really matter why this happens. The fact is that it does. And therefore, you are correct to ask the question about whether this apparently unimportant labelling of the two drinks might, in fact, bias the outcome. The use of these innocuous labels is a great example of a confounding variable, because the preferences that participants express might well be a preference for the letter, rather than a preference for the taste of the drink.
Keep thinking in this way. Keep questioning. Keep evaluating.
Kind regards.'
References
Banyard, P., & Grayson, A. (2008). Introducing Psychological Research (Third). Palgrave Macmillan.
Pfungst, O. (1911). Clever Hans (the horse of mr. Von Osten): A contribution to experimental animal and human psychology. Holt. http://dspace.library.uu.nl/handle/1874/30297
Rosenthal, R., & Fode, K. L. (1963). The effect of experimenter bias on the performance of the albino rat. Behavioral Science, 8(3), 183–189.
Woolfolk, M. E., Castellan, W., & Brooks, C. I. (1983). Pepsi versus coke: Labels, not tastes, prevail. Psychological Reports, 52, 185–186.