Generalization, Replication
& Alternative Explanation

Now that you know the basics of critical thinking, its time to develop some strategies that will help you critically evaluate any research study.  Whenever you come across a study, it is helpful to have a few key questions in mind.  To help you remember these questions, try to memorize this useful mnemonic:  to evaluate a study, think GRAVE!

G stands for Generalization.  Can the results of the study be generalized to a wider population?

R stands for Replicability & Reliability, as well as Research Method.  Can the study be repeated?  Do we get the same results each time?  What are the pros and cons of the research method used in the study?

stands for Alternative Explanations.  Are there different ways of explaining the study's results?

stands for Ecological Validity.  Does the study tell us anything useful about behavior in everyday situations?

stands for Ethics. Did the study follow ethical guidelines?  

Don't worry if this seems confusing right now.  We will look at each of these critical thinking strategies one at a time.  But first, let's start with a research example, so you can see how these skills are applied.
Think Critically

How different are men and women?  In some ways, perhaps a great deal.  Popular stereotypes hold that men are always "trying to get lucky", while women are much more coy when it comes to sleeping with a stranger.  ​Are these stereotypes true?  A Psychology professor named Clark tried to find out.

Read this article on Clark and Hatfield's study .  Make some notes on the following:

1.  What was the aim of the study?

2.  How was the research carried out?  What were the findings?

3.  What was the target population of the study?  In other words, what group of people were Clark and Hatfield interested in understanding?

4.  What was the sample of the study?  What sampling method was used?

If you need to review the terms Target Population and Sample, you may want to quickly read over the page on Sampling Methods
As you read, Clark and Hatfield's study was carried out by observing male and female students at Florida State university as they asked members of the opposite sex for either a date or for sex.  When it came to asking someone out for a date, men and women responded similarly - about half said yes.  But when it came to asking for sex, the results differed dramatically.  Around 75% of men were willing to have sex with an unfamiliar woman, but not a single woman agreed to sleep with a strange man!  These results seem to suggest that men and women are very different, at least when it comes to interest in casual sex.

But how much can we really conclude, on the basis of this study alone?
Generalization

Clark and Hatfield were interested in studying gender differences between men and women.  Hence, the target population of their study is men and women - in other words, pretty much every adult in the world!  However, their sample consisted of just 96 participants, who were all students at Florida State university.

Can we really make conclusions about all of humanity on the basis of such a limited sample?  Perhaps not.  Maybe there is something about American university culture, or maybe even just the culture in Florida State university, that encourages men to seek casual sex, or that sanctions against casual sex for women.  It is conceivable that carrying out this study on a different university campus, or in a different country, or with a different age group, might produce very different results.  

Generalizability is the extent to which a study's findings can be applied to other people - to the entire target population.  Clark and Hatfield weren't just interested in differences between men and women at Florida State university.  They were interested in universal differences between men and women.  

In order for a study to be high in generalizability, it is important for the sample to be representative of the target population.  That means that all of the demographic characteristics of the target population should be included in the sample.  If the target population is all men and women, then a truly representative sample would include men and women from different age groups, different cultures, different occupations, and so on.  The sample should look like a small microcosm of all the men and women in the world.  

Clark and Hatfield's study used a sample that was not representative.  Their sample consisted only of Florida State university students, which fails to account for the diversity in men and women around the world.  Therefore, the study has low generalizability.  Unless more research is carried out with different samples, we shouldn't assume that these results apply for all men and women.


Exam Tip: Evaluating Generalizability

In order to evaluate the generalizability of a study, discuss the following:

1.  What is the target population of the study?

2.  Describe the sample of the study.  Who participated in the study?  What sampling method was used?

3.  Is the sample representative of the target population, or is the sample biased?  To explain why, you could discuss the strengths and limitations of the sampling method

For instance, you might write the following about Clark & Hatfield's study:

"Clark and Hatfield were interested in investigating universal gender differences - hence their target population encompassed all men and women.  However, their sample consisted only of students from Florida State University, who were recruited by opportunity sampling.  The sample does not include men and women from different cultures and age groups, so it is not a representative sample.  The findings of the study - that men are far more interested in casual sex than woman - cannot be generalized to all men and women due to the limitations of the sample.  More research with different samples is needed to confirm Clark and Hatfield's findings."




Alternative Explanations

When psychologists carry out a study, they try to explain their results.  For instance, Clark and Hatfield noticed that men were far more eager to have casual sex than woman.  They explained their results by suggesting that men and women are fundamentally, biologically different when it comes to interest in casual sex.  

However, it is always important to consider whether there are other ways of explaining the results - known as alternative explanations.  Maybe men and women are not so fundamentally different after all - perhaps there are cultural factors at play.  Some argue that Western culture has a sexual double standard - men are taught that its desirable to have lots of sexual partners, while women are taught that sleeping around is shameful.  Perhaps it is this sexual double standard that explains the results.  If men and women were brought up in a culture that treated them equally, then perhaps their interest in casual sex might not be so different, after all.

It is especially important to be aware of alternative explanations when looking at correlational research.  When observing a correlation between two variables, X and Y, many people conclude that X must be the cause of Y.  However, there are other, equally feasible alternative explanations - Y may be the cause of X, or a third factor (Z) may be causing both X and Y, or the correlation may simply be random.  (You may wish to review the page on Correlational Research for more discussion of why a correlation can be interpreted multiple ways).

When there are different possible explanations for a study's results, more research is needed to find out which explanation is correct.  For instance, Clark and Hatfield's study could be repeated in different cultures, including in cultures where men and women are treated more equally, like Sweden, and in cultures where men and women are treated less equally.  This could help settle the debate of whether it is biological differences, or cultural factors, that better explain men and women's willingness to have casual sex.
Exam Tip: Evaluating Alternative Explanations

When evaluating alternative explanations of a study, discuss the following:

1. What conclusions do the researchers draw from the results?

2. Are there any other ways of explaining the results?  In particular, did the researchers consider how both biological and cultural factors might explain the results?

3.  If the study is correlational, what are some different ways of interpreting the correlation?

For instance, you might write the following about Clark & Hatfield's study:

"Clark and Hatfield interpreted their results to suggest that there are fundamental, biological differences that underlie men and women's willingness to have casual sex.  However, cultural differences in how men and women are socialized could also explain the results.  Perhaps men are socialized to regard casual sex as desirable, while women are socialized to regard causal sex as shameful.  More research is needed to investigate how culture may shape sexual behavior in men and women".


Replicability & Reliability

As you should now see, we should be careful about making any firm conclusions from just one research study.  Clark and Hatfield found a huge difference between men and women's willingness to have casual sex with a stranger.  But we still don't know for sure whether that's something that's unique to Florida State university, or something that holds for all men and women.  And we also don't know for sure what's causing the difference - it could be because of biological differences between the sexes, or because of differences in how men and women are treated by society, or both.

How do we resolve these questions?  The answer is simply to carry out more research.  For instance, we could replicate the study with different groups of men and women, repeating the procedure of the study in different universities, with different age groups, even in different countries.  Replicating a study just means to repeat the study again.  If the same results are obtained over many replications, the findings are said to be reliable.

It is considered important for research studies to be replicable.  When researchers publish a study, they are supposed to describe, in minute detail, exactly how they carried out their research.  They should make it possible for anyone to replicate the study, and see if the same results are obtained.  There are, however, a number of exceptions that make a research study difficult or impossible to replicate.

  • Case studies often involve unique, one of a kind situations, and so are often impossible to replicate.  For instance, you learned about the case of David Reinmar, who was born biologically male, but was brought up as a girl after a botched circumcision.  For obvious reasons, we could not repeat this study on another boy.

  • Some famous studies have later been deemed to be unethical - in other words, they should have never been carried out in the first place, due to the harm they inflicted on participants.  It is ethically impossible to replicate these studies, unless significant modifications are made to address ethical concerns
Think Critically

Has Clark and Hatfield's study been replicated?  Are the results reliable?
Read the first half of this article, "Would you agree to have sex with a total stranger?"

1.  Makes notes on at least 3 replications of Clark and Hatfield's research

2.  Based on all the evidence, do you think that differences in male and female attitudes to casual sex are mainly because of biology or because of culture?
One of the strengths of Clark and Hatfield's study is how simple it is.  Anyone could try asking a bunch of strangers for sex, and see if they get the same results.  And in fact, Clark and Hatfield's research has been replicated many times, and the results were found to be reliable.  According to the Psychology Today article, it has been replicated in Denmark (where 59% of single men and 0% of single women said yes to sex with a stranger) and in France (where the numbers were 83% for men and 3% for woman), amongst others.  You can even watch a (non-scientific) attempt to replicate the study below:




















By replicating a study many times, in different cultures, we can investigate whether a behavior is universal (the same all around the world) or culture-dependent (a product of culture).  It seems like no matter what country you study, or what the local culture is, men are more receptive to offers of casual sex from a stranger than women are.  When a human behavior is consistent across very different cultures, biology must be playing a role.

In summary, Clark and Hatfield's study, on its own, raised many questions.  We didn't know whether the results could be generalized to all men and women, or only applied to students at Florida State University.  We also didn't know whether the results were mainly because of biological or cultural differences between the genders.  But after carrying out many replications, and getting reliable results across different samples and different countries, we can now answer those questions - men and women, all around the world, do seem to be very different when it comes to casual sex, which implies there are important biological differences between the genders.
Checklist

  • I can explain what it means to generalize the results of a study

  • I can evaluate the generalizability of a study by considering the makeup of the sample 

  • ​I can consider whether there may be alternative explanations for the results of a study
​​
  • I can evaluate whether it is possible to replicate a study, and whether the results are reliable
​​


Quiz Yourself

1.  Generalizability is the extent to which..

(a) There is a gender difference in the results

(b) The results can be applied to the target population

(c) The results are general, no matter what culture is studied

(d) The results can be applied to all people



2.  To evaluate a study's generalizability, the most important criteria is..

(a) The sample size

(b) The sampling method

(c) An equal number of men and women in the sample

(d) The representative the sample is



3.  An alternative explanation refers to..

(a) A different way of interpreting the results

(b) A result that is culture-specific

(c) A modification in the original study

(d) An explanation that considers both biology and culture



4.  Over the last decade, there has been an increase in young men who don't work, and spend hours each day playing video games.  The researchers conclude that the artificial rewards of video games may make some men uninterested in finding work.  What's an alternative explanation for this finding?

(a) Video game graphics have improved significantly over the past 20 years

(b) Video games may have addictive properties

(c) Video games cause changes in the brain's reward centers, making everyday activities seem less exciting

(d) There are fewer well-paid jobs available to young men



​5.  What is a reason why a study may not be replicated?

(a) The sample of the study did not represent the target population

(b) The study violated ethical guidelines

(c) The study used non-experimental methods

(d) The results of the study are subject to alternative explanations


6.  A cross-cultural study found that people from all over the world were able to recognize a happy person, a sad person, and an angry person by looking at photographs of facial expressions.  What conclusion can be reached?

(a) Facial expressions are determined by the culture in which you grew up

(b) The study is high in replicability, but low in reliability

(c) The study is low in replicability, but high in reliability

(d) There are universal facial expressions related to emotions


Answers

1 - B, 2 - D, 3 - A, 4 - D, 5 - B, 6 - D