Thanks . . .
First of all, thanks to everyone who took part in this exercise. Even though the sample was made up of super-curious market researchers, who were wondering what it was all about, the results proved interesting.
If you weren’t there… The experiment consisted in participants being asked to taste five different-flavoured jelly beans. All participants were given exactly the same selection, and then to record their preferences.
Time to own up!
Not everyone was asked to perform precisely the same task:
- One half of the sample was asked simply to rank five different jelly beans according to taste preference ( 1 = most liked; 5 = least liked)
- The second half was asked to perform the ranking exercise above and to provide reasons for each of their rankings
|Ranking||Ranking alone (n=30)||Rankings with reasons (n=31)|
|1 – most liked||Green||Green|
|3||Yellow speckled||Yellow speckled|
|5 – least liked||Blue||Cream|
The differences in results for the cream and the red beans are statistically significant.
The first thing to note is that, with the exception of the green jelly bean (which appeared first in both arms), and the yellow speckled bean, all the mean rankings differ between the two sets of respondents. In fact, although the yellow speckled bean appears in the middle of each table, its position is different each time, relative to the other beans!
Note, especially, how the assessment of the cream jelly bean, in particular, differed markedly between the two halves of the sample, being ranked 2nd when no reason for the ranking was demanded, but ending up at the bottom of the pile when respondents were required to explain their decision.
So what does this mean?
Although based on a small sample, the experiment on is based on a classic behavioural economics study performed in the early 90s. (Their experiment was based on jam-tasting which, for obvious reasons, didn’t seem sensible in the context of the conference).
What the results suggest is that asking respondents for introspection not only produces rationalisation of response, but can also materially influence the response itself.
This raises really important questions about how we conduct research, whether qualitative or quantitative. It implies that the probing ‘why?’ questions need to be eradicated from our research questionnaires, interview and discussion plans etc. And this is not simply because they are not helpful but, further, because the responses they produce and influence stand a high chance of being misleading.
But what about the green bean then?
The first position of the green bean is interesting. However, what it suggests is that there is something about the green bean that makes it both very likeable in the ‘fast thinking’ mode, and also in the ‘slow thinking’ mode (Kahneman D. Thinking Fast and Slow). In other words, not only do those tasting it ‘simply’ like it because they like it, but they continue to like it even through the process of rationalising their response. Put most simply, it tastes good, and respondents in the ‘reasoning’ group can think of good/acceptable reasons for their preference.
By contrast, the cream bean is well-liked when no explanation is required. However, when having to justify or explain that response, other rational factors are weighed up and this undermines its position. One could speculate as to what those factors might be, but we can’t really be sure. It could be, for example. ‘ the more I think about the appearance of that bean, the more it seems unappealing’ or ‘ it’s pretty sweet-tasting and that does not represent good taste – perhaps I don’t like it as much as I thought I did’. A similar, but less pronounced, difference is seen in the rankings of the red bean.
However, it is not necessarily simply the position of the bean in the ranking table that is of interest here, but where it sits relative to other specific beans. Thus, although the yellow speckled bean appears in the middle of the list in both occasions, the meaning of this position is different each time!
So what does all this mean for me?
First of all, don’t worry: we’re all in the same boat. What it means for all of us is that we are going to have to do some hard (slow, System 2) thinking to change and refine the way we do things. We are not alone – there is a whole big world of research out there, trying to get to grips with this.
(Almost) Last words…
If you took part, we hope you’ve found these results interesting. If you weren’t able to take part, why not try and replicate the experiment among your colleagues: I will be very happy to supply you with the paperwork, but I’m afraid you’ll have to source your own jelly beans – Hardy’s Old Fashioned Sweet Shops are great and, if you speak to them nicely, you might get a discount!
Real last words – Thanks…
To John and Mark at First Line Research for their ace statistical analysis – and I have the tables to prove it!