Threats to a study's validity may be internal or external. Some are controllable, while others are not. External threats to a study's validity include the inability of study results to be generalized to a population or situation other than that studied. Even if a study is internally valid and the demonstrated outcome real, results are not guaranteed to be applicable to other settings. One example is the attempt to extrapolate results obtained with an animal model to a population of patients in the ED. Even in clinical studies, assessing whether the methods described in the setting or circumstances under which the clinical problem typically is encountered is always important. For example, was a patient acuity representative of the norm, or were special resources or consultants used that are usually not available?


Internal threats to validity of a study involve problems with study design or implementation. Bias is the systematic introduction of error, which distorts results of a study in a nonrandom way. Most studies have some potential sources of bias. Researchers are responsible for understanding the influence bias plays in an experiment, minimizing those effects when possible, and identifying potential bias when publishing their results. Bias is entirely different than chance, which is a purely random study outcome. Regardless of how carefully a study is designed, the possibility always remains that demonstrated results are the result of random chance rather than to a real association or cause-effect relationship. A primary purpose of statistical analysis is to estimate the likelihood that results obtained could have occurred solely by chance.

The presence of variables other than those under study, which nonetheless may have had significant effects on the outcome of a study, is a very common research problem. Be alert for these confounding variables, as authors do not consistently identify them.
Study results may be affected to varying degrees simply by the fact that a study is being performed. Physicians who know that they are being measured may be influenced to administer the drug earlier. This tendency of the study situation to artificially influence the outcome of the study is termed the Hawthorne Effect.