Are All Research Participants Outliers?

A Pacific Standard profile of noted social psychologist Joe Henrich has some staggering information about how social scientists conduct their research:

Economists and psychologists, for their part, did an end run around the issue with the convenient assumption that their job was to study the human mind stripped of culture. The human brain is genetically comparable around the globe, it was agreed, so human hardwiring for much behavior, perception, and cognition should be similarly universal. No need, in that case, to look beyond the convenient population of undergraduates for test subjects. A 2008 survey of the top six psychology journals dramatically shows how common that assumption was: more than 96 percent of the subjects tested in psychological studies from 2003 to 2007 were Westerners—with nearly 70 percent from the United States alone. Put another way: 96 percent of human subjects in these studies came from countries that represent only 12 percent of the world’s population.

Here’s more on studies that use American participants:

Among Westerners, the data showed that Americans were often the most unusual, leading the researchers to conclude that “American participants are exceptional even within the unusual population of Westerners—outliers among outliers.”

Given the data, they concluded that social scientists could not possibly have picked a worse population from which to draw broad generalizations. Researchers had been doing the equivalent of studying penguins while believing that they were learning insights applicable to all birds.


MS

This bigger issue with typical university samples is common/well-known enough that there's an acronym to describe it. Western, Educated, Industrialized, Rich, Democratic - WEIRD.

Julie

I think this is especially true of studies about relationships, love, and sex. I think it's ridiculous use university students -- who are generally not in long-term committed relationships and whose prefrontal cortices are still developing -- to extrapolate to the majority of the population.

Seminymous Coward

That's only a problem if they claimed this supposed universal applicability. Proper research articles include information on how the sample was taken.

Incidentally, studies have enough hoops to jump through without trying to ensure the sample includes the Yanomami, Boers, Ainu, Cossacks, and Inuvialuk. There are over 6900 living human languages. Travel is obviously an issue as well.

Just enumerating all the existent combinations of physical sex, gender identity, gender role, sexual orientation, ethnic group, languages spoken, aristocratic class, social class, economic class, educational background, genotypes, and gene expressions (in the epigenetic sense) would be a monumental achievement.

James

I would think there would be as much or more effect from the selection on age, health, and intelligence that comes from selecting one's study participants from university undergrads.

There's also a sub-selection within the undergrad population, as (for most studies) you only get those who're willing to participate in studies. That might, for instance, tend to leave out us unsocial types, students who're too busy holding down jobs, etc.

bob

I think the real mistake made here was pretending that social scientists were actual scientists. Didn't we learn our lesson from political science?

Enter your name...

Well, some of them are, and some of them aren't. There is some good experimental work going on in a few "social science" labs. You'll even find true experimental science—actual randomized, controlled, single-variable experiments—in some business schools. Controlled trials are very useful for determining which advertising approach produces the highest sales for a given product. Google, Amazon, and other highly successful consumer-oriented, internet-based companies run proper experiments constantly. The only difference is that they are designed to discover something about how to extract money from your wallet rather than to discover something about the structure of the physical universe.

Josh

This has been a well-known problem for decades. The reason that this isn't a flaw that negates all social science research is that researchers aren't typically interested in extrapolating their conclusions beyond the demographics of their sample. It is well understood in all social science disciplines that the findings of studies aren't generalizable around the globe to all cultures. Doing research to be so inclusive would be a massive undertaking that would be prohibitive. Convenience samples keep costs in the range that is commensurate with researchers' budgets: minimal. This makes work for researchers around the world to study their culture or subculture of interest and leads to studies with better and more "locally" appropriate designs.