Psychology Research Methods · How p-values and internal validity answer different questions
CORE RULE
The p-value helps us judge whether an observed effect is unlikely to be random variation. Internal validity helps us judge whether the IV caused that effect.
Two questions, not one
Question 1: Is there evidence of an effect? Statistical significance addresses whether the observed difference would be unusual if the null hypothesis were true.
Question 2: What caused the effect? Internal validity addresses whether the IV is the best explanation for the observed change in the DV, rather than a confound, which is a systematic influence.
Example 1 — Significant result + strong internal validity
Students are randomly assigned to two groups. Group A drinks coffee; Group B drinks no coffee. Both groups then complete the same reaction-time test under standardized conditions.
Result: p = .01. The difference is statistically significant for a P-value threshold of .05. Because the experiment is also well controlled and uses random allocation, there is good reason to attribute the difference to caffeine.
Example 2 — Significant result + confound
Two classes are compared. Class A receives a new teaching method; Class B receives the usual method. Class A scores significantly higher on the final test. However, Class A also has a more experienced teacher and because the classes are pre-existing, Class A also happens to be systematically more intelligent than class B.
Result: p = .01. The difference is statistically significant, but the causal conclusion is uncertain. Teacher experience and intelligence are confounding variables and provide alternative explanations for the difference.
A significant result can still come from a study with weak internal validity.
Example 3 — Non-significant result
Participants are randomly assigned to a new memory technique or a control condition. The treatment group scores slightly higher, but the difference is not statistically significant.
Result: p = .32. We do not have strong statistical evidence that an effect exists. Therefore, we should not conclude that the memory technique caused an improvement.
A simple decision sequence
1. Is there evidence of an effect? Look at statistical significance.
2. If there is an effect, what caused it? Evaluate internal validity.
The intuition behind a low p-value
A useful beginner-level intuition is that a low p-value suggests that the observed pattern is difficult to explain as random noise alone. It does not identify the cause of the pattern. Something systematic may be producing the difference, but that systematic influence could be the IV or a confounding variable, or a combination.
SIGNIFICANCE
Is the pattern more than random noise?
INTERNAL VALIDITY
Can we attribute the pattern to the IV?
Technical note
Strictly speaking, p < .05 does not mean that there is less than a 5% probability that the result is due to chance, nor does it mean there is a 95% probability that the IV caused the DV. A p-value is calculated under the assumption that the null hypothesis is true: it describes how unusual the observed result, or a more extreme one, would be under that assumption.
Leave a Reply