Research Methods

• Estimate of systematic variance
• Estimate of error variance
• Calculate a ratio

• F = Between groups var.
• Within groups var.

• Between= variance of group means
• Within= average of variance found within each group

• 1. Systematic variance
• 2. Error or unsystematic variance

Difference due to different levels of I.V.

Differences due to uncontrolled, random factors (not confounding)

• 1. individual differences
• 2. random variations in testing
• 3. measurement error
• 4. experimental treatment not the same for each subject

systematic variation of i.v. should cause systematic variation of the d.v. = systematic variance

differences in group means

group means are not likely to be exactly the same

No systematic var.

that there is a real difference in means or that IV and DV are correlated

would not reduce variance within groups variance

greater than differences within any single group

just due to chance or uncontrolled factors

controlled and uncontrolled factors

• 1. large differences between groups
• 2. small differences within groups

• No, not directly-not in the equation
• Influences whether F ratio is statistically significant

• 1. large differences among group means
• 2. small differences within each group (not confound idea)
• small individual differences
• reliable tests or measures
• consistent application of each level of iv

difference (or variability) in means

a comparison of the influence of different levels of a predictor or independent variable

• Each level of each IV occurs with every other level of all the independent variables
• Uses all possible combinations of levels of 2 or more independent variables

the effect (differences in means) of one iv, averaging,ignoring, or collapsing over the other iv's

• 1. Economy
• 2. Experimental control
• 3. Check generality of the effect of an IV

reduce error variance

the effect of one iv depends on the level of another iv

• main effect of iv A on dv
• main effect of iv B on dv
• Interaction between A and B

• 1. Set up a table of means for each condition
• 2. Draw a figure
• 3. Describe the means for each main effect followed by a description of the results of the ANOVA

• 1. Any t, F, U, orr could be due to chance/luck
• 2. If a result (a, t, F...) occurs by chance infrequently decide result is statistically significant
• 3. Typical reasoning
• 4. examples

often decide that if an outcome is unusual/rare must be due to more than chance.

presumed true unless statistical evidence suggests otherwise

we have information about chance, not how the independent variable works.

Rejuect H_o but it is actually true

Faile to reject the H_o but H_o is false

• 1. Show that changes in Y didnt occur until change in X (temporal precedence rule)
• 2. Show that X and Y are related (covariation rule)
• 3. Rule out other explanations forthe relationship between X and Y (internal validity rule)

• 1. manipulation of X
• 2. Comparison of the effects of various levels of X on Y
• 3. Subjects begin the experiment equivalent on all levels
• 4. control over all other important variables so that all subjects are treated the same except for X (random assignment)

• a. Change in client occurred after the introduction of the program
• b. participating on not participating in program covaries with client success
• c. Rule out other explanations for the relationship between the program and client success
• d. plausible causal mechanisms linking program to client success

greater than 1

around 1

more error variance

• often encountered in quasi experimental research
• Have a single experimental group for which we have multiple observations before and after naturally occuring treatment.
• Instead of observing 1 or 2 3rd grade classes we could observe 3rd grade classes over several years
• We need to know when the time series is interrupted by some treatment
• Then we compare observations beforea nd after the treatment to see whether it had any effect.

• Source of causation;
• baseline condition maturation
• history
• selection bias

• Deviant case analysis (a non equivalent control)
• detective work

• changes in participants and environment
• delayed effects

• nonequivalent control group
• detective work

• dimensions on which to match
• regression artifacts

• matching
• include true independent variable
• see interactions

• Confoundings with time of testing
• generation of birth

• Cross sequential design
• include a true i.v. & seek interaction
• Converging operations

regression artifacts

interaction

• Significance level
• it does notindicate repeatability

Calculating within groups variance (variance within each group pooled)

• range of group means
• (range of scores in gp 1+ gp 2+gp 3 etc..
• # of groups

• 1. Systematic variance
• 2. Error or unsystematic variance

• a simple inferential statistic that can be used in place of a t test.
• It can be used in many instances in which you have tested twoindependent groups of subjects

• 1. put the scores in order from smallest to largest
• 2. For each score in the group w/ the lower mean, count the # of scores that are smaller in the other group. Thus you'll have 1 # for each score in the group with the lower mean that tells you how many scores are smaller in the other group than that score\
• 3. Addthese numbers together
• 4. For each score in the group with the lower mean, count the # of scores in the higher group that are tied or are the same. Add these ties together and divide by 2
• 5. Add theresults of C and D together to obtain the mann whitney u

• Is a particular program actually delivering the services it is designed to deliver?
• Is a law having the desired effect?

• a. Change in client occurred after the introduction of the program
• b. Participating or not participating in the program covaries with client success
• c. Rule out other explanations for the relationship between the program and client success
• d. Plausible causal mechanisms linking the program to client success

• One shot case study
• There is no comparison
• Descriptive research

• Does it answer...
• Change in the client occurred after intro to program
• Having or not having program covaries with client success
• Rules out other explanations for relationship between program and client success

• Extremely low internal validity
• Cant even show that the client changed during the program

interrupted time series

• pretest post test design (before-after) o x o
• Potentially suffer from all the threats of within subjects design
• change occurred after intro to program
• Program/no program varies with client success
• rule out other explanations for the correlation b/w program/noprogram and client success

one shot case study

Pret test post test

Static Group Comparison (ex post facto)

Pretest posttest control

Might manipulate X, but dont compare groups that are formed based on random assignment

More opportunity to discredit alternative interpretations of data

• Add observations
• 1. Additional times before and after program introduced-time series studies
• OXO -> OOOOOOOXOOOOOO
• 2. Additional people who havent received the program (non equivalent control group - no random assignment - STATIC GROUP COMPARISON
• OXO -> OXO
• O-O (non equivalent control group

Get many measures before and after some natural or planned intervention

add a comparison group

Increase number of observations, add comparison group or both

• 1. comparison and "program" group have same amount of time to mature
• 2. history should influence both groups equally
• 3. testing should influence both groups equally

• Finding a good comparison group...
• Cant randomly assign-between groups variable: ex post facto

groups different because of the way they were selected or assigned to groupss

• 1. no random assignment
• 2. quasi-comparison using additional measures before and after or non equivalent control groups
• 3. moderate control
• 4. often field based
• 5. may never be able to eliminate confounding variables