Ecological Validity &
Research Method

Remember GRAVE?  So far, we have looked at evaluating the Generalizability, Replicability & Reliability, and Alternative Explanations of a study.  On this page, we'll be examining what we mean by Ecological Validity and the R​esearch method used in a study.
Video Activity

In July 2016, in Minnesota, a man named Philandro Castille was driving with his girlfriend and young daughter.  Philandro worked as a supervisor at a local high school.  Noticing a broken taillight, a police officer pulled Philandro over, and instructed Philandro to show his driver's license and registration.  As Philandro reached for his license, he informed the police officer that he was carrying a gun, which he was licensed to carry.

Those were the last few minutes of Philandro's life.  The police officer shot Philandro seven times, in front of his girlfriend and daughter.

Watch the video below, which was taken by Philandro's girlfriend, moments after the shooting.  Why do you think the police officer shot Philandro?  Do you think that race played any role in the events that occurred?
Few people believe that the police officer wanted to kill an innocent man.  And yet, longstanding questions about race, prejudice, and police conduct intensified around Philandro's death.  Did the police officer assume that Philandro was dangerous just because he was black?  Would he have killed a white person under the same circumstances?  The Governor of Minnesota wondered, "Would this have happened had those passengers would have been white?  I don't think it would have."

After the shooting of Philandro, many pointed to the role that implicit associations may have played in the events.  Implicit associations are thoughts and feelings, outside of conscious awareness, that may affect how we think about (and act towards) different groups of people.  For instance, if you associate tall people with leadership, you may be more likely to pay attention to a teacher or boss who has an imposing stature.  To find out more about implicit associations, you are now going to take an implicit association test yourself.
Try it Out: The IAT

1.  Go to Project Implicit , and select the Weapons IAT (the fourth from the top)

2.  Describe the Weapons IAT in your own words.  What does it measure?

3.  Most Americans seem to implicitly associate Black faces with weapons.  Can the results of the IAT explain the police shooting of Philandro Castille?  Explain why, or why not.
Ecological Validity

The Weapons IAT has found that most Americans have a faster reaction time when Black faces are paired with weapons.  This suggests that people implicitly associate Black people with violence and danger - and that this might explain why, in some cases, police officers mistakenly kill innocent black men.

Other people, however, disagree.  They claim that the IAT doesn't really tell us anything significant.  After all, the IAT involves sitting in front of a computer, and pressing keys on a keyboard.  So what, they say, if your reaction time is slightly faster, or slower, on certain tasks.  This isn't real life; it's an artificial situation.  How police officers behave in real life, they say, is completely unrelated to results on the IAT.

Ecological validity refers to the extent that results from a study can be generalized to real-life settings.  Psychology experiments are often very different from real-life situations.  It would be very interesting if we could design an experiment that would involve studying police officers in simulated life-or-death situations, but of course that would be impossible.  We can't (ethically) create or observe these situations in the same way that they would happen in real life.  So Psychologists design an experiment, such as the IAT, that involves carefully controlled variables that are easy to manipulate and measure - but which (arguably) may not tell us very much about behavior in the real world.

The debate over the IAT continues on.  Some psychologists argue that the IAT provides credible evidence that police officers are biased to use deadly force against blacks, while critics say that the IAT lacks ecological validity, and the results of the test are basically meaningless.  At the moment, there is not enough evidence to settle the debate.  Although the results of the IAT are fascinating (and shocking), there are certainly valid questions about whether the IAT is an ecologically valid measure of police shootings.  (To learn more about some of the research on this issue, take a look at this article from the PBS)


Exam Tip

Many Psychology students tend to evaluate lab experiments by writing "since this experiment took place in a lab, it has low ecological validity".

This is not strong critical thinking.  In order to evaluate the ecological validity of a study, you first need to identify the real-life behavior that is the subject of investigation.  Then, consider whether the experiment involves a realistic simulation of what happens in real life, or whether the experiment involves a high degree of artificiality.  If the study is indeed highly artificial, you could argue that the results are not generalizable to real-life situations.

For instance,

"The IAT has been used to explain police shootings of innocent black men.  However, some argue that measuring reaction times on the IAT reflects an artificial task, and does not necessarily predict how a police officer reacts in the real world.  If so, then the IAT could be said to have low ecological validity.  More research is needed to confirm whether there is a link between reaction times on the IAT and the use of police force in real-life situations"
Research Method

When evaluating a study, it is always a good idea to identify and describe the research method, and to discuss the strengths and limitations of this methodology.  For instance, when discussing an experiment, you could identify the independent and dependent variables, and explain how a cause-and-effect relationship has been established.  You could also discuss the sampling method, or point out any concerns with demand characteristics and ecological validity.  You may want to review the page on Experiments , then try to answer the questions below.
Try it Out

See if you can answer the following questions about the methodology of the IAT.  Scroll to the bottom of the page for answers.

1.  What is the independent and dependent variable in the IAT?

2.  What sampling method was used?  What is a strength and limitation of this sampling method that is relevant to the IAT?

3.  What sort of experiment is the IAT?  What is a strength and limitation of this type of experiment?
Checklist

  • I can evaluate the ecological validity of a research study, justifying my answer based on the similarity of the study's conditions with the real-life behavior under investigation

  • I can discuss the research method of a study, for instance, by identifying key features of the methodology, and evaluating its strengths and limitations
Quiz Yourself!

1.  After the death of Philandro Castille, the governor of Minnesota questioned whether a white person, in the same situation, would have been killed by the police.  What is the most likely interpretation of the governor's statement?

(a) Because of implicit bias, some police officers deliberately kill unarmed black people.

(b) Because of implicit bias, a police officer is more likely to show prejudice on the IAT.

(c) Because of implicit bias, a police officer is more likely to associate black people with weapons, leading to grave errors in judgement.

(d) Because of implicit bias, a police officer is more likely to associate white people with harmless tools, leading to grave errors in judgement.


2.  The IAT Weapons test found that...

(a) Most Americans have faster reaction times to white faces than to black faces

(b) Most Americans have faster reaction times to weapons than to harmless objects

(c) Most Americans have faster reaction times when white faces and weapons are paired together

(d) Most Americans have faster reaction times when white faces and harmless objects are paired together


3.  Ecological validity refers to...

(a) Whether the results of a study can be generalized to real life situations

(b) Whether the results of a study are confirmed by multiple researchers

(c) Whether the validity of the study is harmful to the environment

(d) Whether the validity of the study shows a clear link between the IV and DV


4.  Some criticize the ecological validity of the IAT.  On what grounds is this criticism made?

(a)  The IAT did not control all extraneous variables

(b) The IAT is measured on a computer, rather than in a police academy

(c) Reaction times on a computer-based task may not generalize to real life situations

(d) The IAT did not include police offices of a variety of ethnic backgrounds
Answers

Try it Out

1.  The IV is the pairings (white & harmless objects, black & weapons OR white & weapons, black & harmless objects) and the DV is reaction time

2.  Self-selected sample.  An advantage is that participants will be motivated to try their best on the IAT, while a disadvantage is that participants may be different from the general population. (For instance, they may be particularly aware of race issues, or particularly interested in exploring questions of prejudice)

3.  The IAT is a lab experiment.  Strengths include high control over experimental conditions and high replicability, while limitations include demand characteristics (participants may guess what the experiment is about, and change their behavior accordingly) and limited ecological validity (the results may not be generalizable to real life law enforcement situations)

Quiz Yourself

1 - C, 2 - D, 3 - A, 4 - C