Card Set Information
Eval qualifying exam
for the quals
Threats to Validity
Threats to Statistical Conclusion Validity
Low Statistical Power
Violated Assumptions of Statistical Tests
Fishing and the Error Rate Problem
Unreliability of Measures
Restriction of Range
Unreliability of Treatment Implementation
Extraneous Variance in the Experimental Setting
Heterogeneity of Units
Inaccurate Effect Size Estimation
SCV: Low Statistical Power
An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant.
SCV: Violated assumptions of Statistical Tests
Violations of statistical test assumptions can lead to either overestimating or underestimating the size and significance of an effect.
SCV: Fishing and the Error Rate Problem
Repeated tests for significant relationships, if uncorrected for the number of tests, can artifactually inflate statistical significance
SCV: Unreliability of Measures
Measurement error weakens the relationship between two variables and strengthens or weakens the relationship among three or more variables.
SCV: Restriction of Range
Reduced range on a variable usually weakens the relationship between it and another variable.
SCV: Unreliability of Treatment Implementation
If a treatment that is intended to be implemented in a standardized manner is implemented only partially for some respondents, effects may be underestimated compared with full implementation.
SCV: Extraneous Variance in the Experimental Setting
Some features of an experimental setting may inflate error, making detection of an effect more difficult.
SCV: Heterogeneity of Units
Increased variability on the outcome variable within conditions increases error variance, making detection of a relationship more difficult.
SCV: Inaccurate Effect Size Estimation
Some statistics systematically overestimate or under estimate the size of an effect.
Ways to increase statistical power
use matching, stratifying, or blocking
measure, and correct for covariates
use larger sample sizes
use equal cell sample sizes
improve measurement (e.g. increase range of measurements/reduce dichotomized variables, add additional waves of measurement)
increase the strength of treatment (e.g. increase dose differential)
Use a within-participants design
Use homogeneous participants selected to be responsive to treatment
Reduce random setting irrelevancies
Ensure that powerful statistical tests are used and their assumptions are met. (e.g. transforming the data)
Internal Validity Definition
Inferences about whether observed variation between A and B reflects a causal relationship from A to B in the form in which the variables were manipulated or measured.
Threats to Construct Validity
Inadequate Explication of Constructs
Confounding Constructs with Levels of Constructs
Treatment Sensitive Factorial Structure
Reactive Self-Report Changes
Reactivity to the Experimental Situation
Novelty and Disruption Effects