Join | Login





      

Rotem Kahalon  
 
Verena Klein  
 
Inna Ksenofontov  
 
Johannes Ullrich  
 
Stephen C. Wright  

Lack of sample diversity: How biased is our research practice?

Rotem Kahalon, University of Zurich, Switzerland 
Verena Klein, University of Michigan 
Inna Ksenofontov, Osnabrück University, Germany 
Johannes Ullrich, University of Zurich, Switzerland 
Stephen C. Wright, Simon Fraser University, Canada

Imagine scrolling through your google scholar notifications trying to get a quick update of recently published articles in your field. Based on the article titles, which publications do you thinks would be most likely to inspire further consideration? Our research suggests that researchers may pay less attention to articles if the title mentions the country that the sample was drawn from, and that this is especially true if that country is not the United States.

The problem

The academic literature in psychology lacks diversity in terms of the studied populations. Most research includes participants from Western, educated, industrialized, rich, and democratic (WEIRD) countries. In fact, most is done in the U.S., and is authored by U.S. scientists. As a result, much of the current psychological knowledge is actually based on only 11–15% of the world’s population and really cannot be considered representative of humanity as a whole. 

However, the problem is not just the paucity of published non-WEIRD research but also the lack of attention to and citation of research in non-WEIRD contexts which further reduce its impact on the field.  

Although we would like to think that our findings are relevant and generalizable to all humans, it is clear that social psychological phenomena cannot be well defined independent of their context, and overlooking cultural influences seriously jeopardizes our understanding of these phenomena.

It also leads us to perceive and treat WEIRD populations, and especially the U.S., as the norm for human behaviour, while other populations are perceived as unique and therefore deviant from the norm (i.e., special cases).

Practices that contribute to this problem

In our own work we investigated how mentioning the sample’s country in the article title might contribute to the problem.

Analyzing current publication practice of four leading social psychology journals (Study 1) and conducting two experiments (Study 2), we found that mentioning the sample’s country in the title is more prevalent in research articles that include samples from non-WEIRD countries. Crucially, this practice was associated with less scientific attention and fewer citations – reasonable proxies for impact and inclusion.

We suggest that this phenomenon represents a (perhaps unintentional) form of structural discrimination, which can lead to underrepresentation and reduced impact of social psychological research done with non-WEIRD samples, and reduce the profile and opportunities of those doing research with these populations.

In addition, citing articles that include only WEIRD samples, even when articles with non-WEIRD samples are available, perpetuates the status quo and further solidifies WEIRD populations’ place as the prototype for human psychology.

What can we do about it? 

How can we make the core literatures in psychology more inclusive and representative of all human behaviour?

First, the field as a whole, including authors, reviewers, and editors must make efforts to make psychological science less WEIRD.

Be aware - We must be more aware of subtle mechanisms that affect our decision making. Given that mentioning the sample’s country in the title reduces the probability that the article will get scientific attention, please don't ask (only) authors from “non-WEIRD” countries to state the country in the title if you serve as an editor or a reviewer. Also, authors who do research outside the U.S. should consider the possible negative outcomes when stating the studied population in the title.

Guidelines - Ideally, all authors should be held to the same standards and highlight the origin of their samples, since a complete understanding of psychological research requires the consideration of context. However, based on our results we suggest that journal guidelines should require a description of the sample, including country, in the abstract but not in the title.  

For Further Reading: 

Cheon, B. K., Melani, I., & Hong, Y. Y. (2020). How USA-centric is psychology? An archival study of implicit assumptions of generalizability of findings to human nature based on origins of study samples. Social Psychological and Personality Science, 11, 928–937. https://doi.org/10.1177/1948550620927269 

Kahalon, R., Klein, V., Ksenofontov, I., Ullrich, J., & Wright, S. C. (2021). Mentioning the sample’s country in the article’s title leads to bias in research evaluation. Social Psychological and Personality Sciencehttps://doi.org/10.1177/19485506211024036

Rad, M. S., Martingano, A. J., & Ginges, J. (2018). Toward a psychology of Homo sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences of the United States of America, 115, 11401–11405. https://doi.org/10.1073/pnas.1721165115 

back to menu