How we change what others think, feel, believe and do

Experimental Effect

Explanations > Social Research > Statistical principles > Experimental Effect

Description | Example | Discussion | See also

Description

The effect, or effect size, is an indication of the practical importance of an experimental result.

In essence, 'effect' is the gap between two measures, although is must be measured with a statistical value. a big effect means the two measures are very different, not just 'different' (which is what 'statistically significant' means).

Experimenters thus seek not only statistical significance but also a large effect. Sadly, they do not always find both in the same place.

Measures

The most common measure of effect is the Pearson correlation, r.

r = SQRT( SS_M / SS_T)

Where SS_M is the between-groups sum of the squares, and SS_T is the total sum of squares.

A slightly more complex measure used to reduce bias due to sampling (as opposed to using a population) is omega, w. This is calculated as:

w = SQRT( (MS_M - MS_R) / (MS_M + ((n-1) x MS_R)) )

Where MS_X is the mean sum of the squares (SS/df), with M being between groups and R being within groups. 'df' is the degrees of freedom.

Omega is typically used for ANOVA effect calculation (where MS is already used in the F-ratio).

Effect size

Cohen (1988) gives the rules of thumb for effect size for r (or omega):

Small effect: r = 0.1 (the effect explains 1% of the variance)
Medium effect: r = 0.3 (the effect explains 9% of variance)
Large effect: r = 0.5 (the effect explains 25% of variance)

Note that r is non-linear, and doubling it does not double the effect size.

Example

A measure of men and women in a study shows a statistically significant difference in their body mass index (BMI). r is calculated as 0.05 which is not that big an effect and considered not worth reporting.

However, when the data is segmented based on age, it is found that r is 0.43. The statistical significance is not as high, but the effect is much greater than for gender difference and considered to be carried forward as the main findings of the report.

Discussion

An experiment can report a statistically significant result, but this just says that the experiment has shown that there is a difference between two conditions, not that it is 'significant' in the usual sense of meaning 'big' and 'important' or otherwise earth-shattering in any way.

It is important in reporting experiments to indicate the effect as well as the significance. The American Psychological Association (APA) now recommends that all experimental reports include an indication of effect.