Meta Analysis

Meta-Analysis of Correlations
By. Dr. Andy Field

Meta analysis is a statistical technique by which information from independent studies is assimilated. Traditionally, social science literatures were assimilated through discursive reviews. However, such reviews are subjective and prone to reviewer-biases such as the selective inclusion studies, selective weighting of certain studies, and misrepresentation of findings (see Wolf, 1986). The inability of the human mind to provide accurate, unbiased, reliable and valid summaries of research created the need to develop more objective methods. Meta-analysis arguably provides the first step to such objectivity (see Schmidt, 1992), although it too relies on subjective judgments regarding study inclusion (and so is still problematic because of biased selections of studies, and the omission of unpublished data-the file drawer problem). since the seminal contributions of glass (1976), Hedges and Olkin (1985), Rosenthal and Rubin (1978) and Hunter, Schmidt and Jackson (1982) there has been a meteoric increase in the use of meta analysis. Field (2001) reports that over 2200 published articles using or discussing meta-analysis were published between 1981 and 2000. of these, over 1400 have been published since 1995 and over 400 in the past year. Clearly, the use of meta-analysis is still accelerating.

Basic Principles

To summarise, an effect-size refers to the magnitude of effect observed in a study, be that the size of a relationship between variables or the degree of difference between group means. There are many different metrics that can be used to measure effect size: the Person product moment correlation coefficient, r; the effect-size index, d; as well as odds ratios, risk rates, and risk differences. of these the correlation coefficient is used most often (Law, Schmidt, & Hunter, 1994) and so is the focus of this study. Although various theorists have proposed variations on these metrics (for example, Galss's delta, cohen's d, and Hedges's g are all estimates of Thou), conceptually each metric represents the same thing: a standardized from of the size of the observed effect. Whether correlation coefficients or measures of differences are calculated is irrelevant because either metric can be converted into the other, and statistical analysis procedures for different metrics differ only in how the standard errors and bias correlations are calculated (Hedges, 1992).

In meta-analysis, the basic principle is to calculate effect sizes for individual studies, convert them to a common metric, and then combine them to obtain an average effect size. Studies in a meta-analysis are typically weighted by the accuracy of the effect size they provide (i.e. the sampling precision), which is achieved by using the sample size (or a function of it) as a weight. Once the mean effect size has been calculated it can be expressed in terms of standard normal deviations ( a Z score) by dividing by the standard error of the mean. A significance value (i.e. the probability, p, of obtaining a Z score of such magnitude by chance) can then be computed. Alternatively, the significance of the evarage effect size can be inferred from the boundaries of a confidence interval constructed around the mean effect size.

Johnson, Mullen and Salas (1995) point out that meta-analysis is typically used to address three general issues: central tendency, variability and prediction. Central tendency relates to the need to find the expected maginitude of effect across many studies (from which the population effect size can be inferred). this need is met by using some variation on the average effect size, the sinificance of the this average or the confidence interval around the average. The issue of variability pertains to difference between effect sizes acrosss studies and is generally adressed using tests of the homogenity of effect sizes. The question of prediction relates to the need to explain the variability in effect size across studies in terms of moderator variables. This issue is usually addressed by comparing study outcomes as a function of differences in characteristics that vary over all studies. As an example, differences in effect sizes could be moderated by the fact that some studies were carried out in the USA whereas others were conducted in the UK.