June 25, 2014 by comparativelypsyched
Humans are not rational agents. Everyday, we commit a panoply of logical fallacies, taking mental short-cuts to ease our cognitive burden and limit injuries to our egos. We prefer small immediate payoffs to larger future ones, spending hours on social networking sites while putting off pressing commitments that will lead to a better grade, job, or pay-rise. When something goes wrong we attribute the cause to an external force we had no control over, while positive developments are likely our own doing. But surely we reign in our irrational side when pursuing objective truth? Unfortunately, a recent study has found that it is too easy for our human biases to affect scientific research and it seems not enough is being done to prevent these errors from occurring.
When recording an animal’s behaviour, researchers will code that behaviour in real time or from recorded video or audio footage. Imagine you were studying scratching behaviour in capuchin monkeys. Why scratching? Because we know primates scratch themselves more often when feeling anxious (Polizzi di Sorrentino, Schino, Tiddi, & Aureli, 2012). While studying these monkeys in a zoo enclosure, all you have to do is record whenever a monkey scratches themselves (as conscientious scientists we have defined this as the repeated movement of the fingertips over a piece of skin). While watching one of your subjects you see their hand dart to their leg for a fraction of a second. Was that a scratch? You didn’t get a good view, and it happened so quickly. While you may not be 100% positive, you are certain enough to decide to note that a scratch took place. It is during cases like this, these “maybe behaviours”, that error can be introduced into our objective observations. Humans will see what we expect to see, and it is rare that a scientist will begin a research project without expecting to find a particular result. With scratching behaviours we may have been interested in whether monkeys scratch more (i.e. feel more anxious) when they are alone or with the rest of their group. If we had initially assumed they would be more comfortable in a group, when faced with a “maybe scratch” we might be more likely to count it when the monkey is alone. Does this really happen?
Dr. Frank Tuyttens and his colleagues examined this observer bias in detail. They asked undergraduate students to watch videos of animals and record certain behaviours. Beforehand, the students were told what behaviours to expect under certain circumstances. For example, in one study the students were told that pigs of higher breeding value would demonstrate less negative social behaviours and more positive ones. They were then presented with a series of video clips and were explicitly told which pigs were of a higher breeding value. In reality, both high and low value pigs were the same, the recordings were just manipulated to give the impression that they were different (videos were mirror flipped and the colour or contrast was altered). Even though they were watching the same videos, students rated the pigs differently depending on their expectations; the pigs thought to be of better stock appeared to have more positive social interactions and less negative ones.
In a second study, after being told that cows pant more intensely when it is warmer, students were asked to record the severity of panting in a number of subjects. They were again asked to analyse recorded footage and this time the ambient temperature at the time the video was recorded was available to view in the corner of each video. To test whether this information would influence the students’ observations, the experimenters presented some videos twice, one was the original and one contained falsely elevated temperature information. Again, even though the students were watching the same videos, they “observed” more intense panting in the animals housed in conditions they thought to be warmer.
But surely scientists recognise their limitations and take steps to limit the effects of these biases. Tuyttens and his colleagues looked at some of the most highly regarded academic journals publishing research on animal behaviour. While the best way of preventing observer bias is to have behaviours coded by someone unaware of the predictions of the study, only 9-16% of articles were taking this most stringent measure. These findings suggest not enough researchers are taking the extra precautions to validate their findings. It could be that the procedures used in many of these studies don’t allow the use of an unbiased observer. For example, while watching a capuchin monkey in a zoo we cannot be blind to whether there are other monkeys nearby. Under these circumstances the relevant behaviours can be coded by two raters, however, it is not clear whether a similar problems would persist. While this practice might limit our margin of error, if one coder can be tricked, won’t a second be fooled too?
There is no easy solution to this issue but it is important that researchers agree that the current standards are not good enough. It is the responsibility of researchers to improve the reliability of their data and for reviewers of this data to insist on bias-free behavioural coding as a condition of publication. Confidence in published scientific research should be a priority and it seems that advancing the quality of the procedures is certainly easier than advancing the quality of the irrational observers.
Polizzi di Sorrentino, E., Schino, G., Tiddi, B., & Aureli, F. (2012). Scratching as a window into the emotional responses of wild tufted capuchin monkeys. Ethology, 118(11), 1072–1084. doi:10.1111/eth.12008
Tuyttens, F., de Graaf, S., Heerkens, J., Jacobs, L., Nalon, E., Ott, S., Stadig, L., Van Laer, E., & Ampe, B. (2014). Observer bias in animal behaviour research: can we believe what we score, if we score what we believe? Animal Behaviour, 90, 273-280. doi:10.1016/j.anbehav.2014.02.007