Prepare a Validity Paper
BTM 8103, Assignment 2
DuBose, Justin Z.
Dr. Robert Levasseur
20 March 2018
Validity in research is one of the most important aspects in research design (Mohajan, 2017). Indeed, scholars may discredit even the most compelling research if steps are not taken by the researcher to ensure validity in their design, methodology, analysis, and conclusions.
One of the reasons validity is so important in research design and methodology is that it not only brings about transparency and limits implicit bias on the part of the researcher, but it also constitutes the best practice of the researcher (Tuval-Mashiach, 2017). In other words, when the researcher ensures validity in their design, they are communicating with everyone else that their conclusions are trustworthy and, therefore, credible.
This paper will examine three primary types of validity: external validity, internal validity, and construct validity. Additionally, this paper will discuss the threats to validity and their potential impact upon the envisioned research of the author.
External validity is a form of research validity which addresses the degree to which research findings can be accurately projected onto other contexts and populations (Cozby, 2014 p. 73). Scholars have identified several key questions that researchers should answer when designing and constructing external validity into research. Some of these questions include: “Can the results be replicated with other operational definitions of the variables? Can the results be replicated with different participants? Can the results be replicated in other settings?” (Cozby, 2014, p. 89).
When researchers answer these questions, they can confidently generalize their research findings and others can trust their generalizations (Trochim, Donnelly, & Arora, 2010, p. 83). For example, if a researcher can ensure randomization of participants in the study, then this validates the results to larger populations and contexts. Thus, a major benefit of meticulous safeguarding of external validity is the generalization of research findings to area beyond the research itself.
While external validity pertains to projecting research findings onto other contexts and populations, internal validity refers to the “accuracy of conclusions about cause and effect” (Cozby, 2014, p. 73). As researchers conduct research on various subjects, selecting various participants, and examining various relationships, internal validity ensures that their conclusions about relationships between these variables are accurate. As this is an important component of any study, it is imperative that researchers understand how to ensure internal validity in their design. Cozby (2014) noted that, “a study has high internal validity when strong inferences can be made that one variable caused changes in another variable” (p. 87).
Cozby (2014) noted that experiments (as opposed to surveys, interviews, questionnaires, etc.) are more likely to ensure high internal validity in a study. The reason for this correlation is due to the high degree to which the researcher can closely observe the effect one variable has on another. The researcher can also alter “temporal precedence” in an experiment (Cozby, 2014, p. 87). Causal variables can be re-ordered and re-examined by the researcher in order to more closely observe the cause-and-effect relationship between variables. Therefore, researchers concerned with maintaining a high degree of internal validity will place controls on their research experiment for the purpose of more closely and accurately observing causal relationships.
Construct validity refers to the design and methodology of research and the degree to which that design accurately studies and measures variables (Cozby, 2014, p. 73). One way to consider construct validity is to think of the “adequacy of the operational definition of variables” (Cozby, 2014, p. 75). In other words, the question for the researcher to ask and answer to ensure high construct validity is: “Does the operational definition of a variable actually reflect the true theoretical meaning of the variable?” (Cozby, 2014, p. 76). Perhaps Trochim, Donnelly, and Arora (2016) offer a simpler definition of construct validity. They define construct validity as “the degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations are based” (p. 28).
Researchers concerned with maintaining high construct validity will ask and answer the question of their research: “Did you implement the program you intended to implement, and did you measure the outcome you wanted to measure?” (Trochim, Donnelly, & Arora, 2010, p. 28). One major danger is that researchers design their study to measure one variable but, in fact, measure a different variable entirely. One fictitious example may be a study constructed to measure the correlation between hours of study by tenth graders and geometry scores. Researchers may conclude from this fictitious study that the less time a tenth-grade student spends studying geometry, the higher their test scores. However, they may fail to note in their study that these tenth-graders were utilizing digital study tools which required less time than traditional study methods. Therefore, the variable they actually measured (the impact of digital study tools) would be different than the variable which they intended to measure (study time).
How validity impacts research
The envisioned research which I intend to undertake deals with the field of e-leadership. I envision researching the effects of periodic personal interaction by the e-leader with individual virtual team members on overall team cohesion and performance. In this envisioned research, I will seek to examine three types of personal interaction between e-leader and virtual team members – face-to-face meetings, professional development, and individual coaching sessions – and their impact on team cohesion and performance.
The design of this envisioned research is in the form of a field experiment. The envisioned participants in this study are employees of a multi-billion dollar financial services company who are on virtual teams dispersed over a ten state area. While they conduct regular, virtual meetings to discuss sales numbers, this field experiment would introduce a new variable of individual meetings between e-leader and team members and study the effect of these meetings on team cohesion and performance. As previously discussed, while field experiments have certain advantages when it comes to ensuring validity, they also present unique threats to the research.
Field experiments, for example, lack the research advantage of experimental control (Cozby, 2014, p. 90). In a controlled environment, the researcher maintains close control of research variables so that cause and effect relationships can be more accurately observed. However, in a field experiment, the researcher observes cause and effect in a natural environment. One example is a study in which college students passed a confederate who either sneezed, coughed, or had no reaction as they passed (Lee, Schwarz, Taubman, & Hou, 2010). The students were then asked to complete a survey regarding their perception of being at risk for the flu. Results consistently concluded that those who passed when the confederate either sneezed or coughed perceived being at greater risk for “contracting a serious disease, having a heart attack prior to age 50, and dying from a crime or accident” (Cozby, 2014, p. 90).
The dilemma in this field experiment is that researchers have no control over participants and their level of education, race, gender, field of employment, and many other factors which could be more closely controlled in a designed experiment. Similarly, my envisioned research would lack certain controls which could provide greater validity as it pertains to cause-and-effect relationships. The researcher would have no control, for example, over the participants as they are already selected and employed on these virtual teams.
Cozby (2014) identified two great threats to ensuring validity in field experimentation. He noted that “(1) it can be difficult to determine the direction of cause and effect and (2) researchers face the third-variable problem – that is, extraneous variables may be causing an observed relationship.” (Cozby, 2014, p. 82). In this envisioned research, the threat to external validity is that the financial service industry limits generalization. Thus, any steps taken to ensure the greatest randomization of virtual team participants would increase external validity. Similarly, the greatest threat to internal validity is not accounting for a certain variable which may be present during meetings between the e-leader and virtual team members. For example, is music playing in one type of meeting and absent in another? Is the type of room or furniture arrangement in the room impacting the interaction between e-leader and virtual team member? The researcher must take steps to limit the presence of other variables to ensure high internal validity. Likewise, the greatest threat to construct validity is that the study actually define and measure “team cohesion” and “team performance” and not some other variable, such as “employee satisfaction” or “e-leader effectiveness.” Steps taken here could include aligning a definition of both cohesion and performance with the corporate standards of the financial services industry as well as existing research in the field. This step would ensure high construct validity within the study.
Cozby, P. C. (2014). Methods in behavioral research (12th ed.). Boston, MA: McGraw Hill Higher Education.
Lee, S. S., Schwarz, N., Taubman, D., & Hou, M. (2010). Sneezing in times of a flu pandemic: Public sneezing increases perception of unrelated risks and shifts preferences for federal spending. Psychological Science, 21, 375–377. doi:10.1177/0956797609359876
Mohajan, H. (2017). Two criteria for good measurements in research: Validity and reliability. Annals of Spiru Haret University Economic Series, 17, 59-82. doi:10.26458/1746
Trochim, W., Donnelly, J., & Arora, K. (2016). Research methods: The essential knowledge base (2nd ed.). Mason, OH: Cengage.
Tuval-Mashiach, R. (2017). Raising the curtain: The importance of transparency in qualitative research. Qualitative Psychology, 4(2), 126-138. doi:10.1037/qup0000062
NG, LR, NCU, USAR
My collection of personal papers written over the years