We use cookies to give you the best experience possible. By continuing we’ll assume you’re on board with our cookie policy

Multiple Effect Sizes Essay Sample

The whole doc is available only for registered users OPEN DOC

Get Full Essay

Get access to this section to get all help you need with your essay and educational issues.

Get Access

Multiple Effect Sizes Essay Sample


Main and Moderator studies

For our reviews, we are faced with the issue that multiple effect sizes are reported from the same sample of participants (e.g., due to various outcome variables such as reaction time and error rates or due to multiple contrasts stated on the same outcome variable). This introduces statistical dependencies between effect sizes from the same study which violates the independence assumption of classical meta-analysis (e.g., Hedges & Vevea, 1998). Ignoring these dependencies can lead to unacceptably low coverage probabilities and increased type-1-error rates for primary and moderator analyses, respectively (Van den Noortgate, López-López, Marín-Martínez, Sánchez-Meca, 2013; López-López, Van den Noortgate, Tanner-Smith, Wilson, & Lipsey, 2017, respectively).

One frequently used approach to handle this issue is the strategy of averaging effect sizes for each independent sample. While this results in independent effect sizes, it leads to underestimations of between-study heterogeneity (Cheung & Chan, 2004) and is associated with reduced statistical power (López-López, Page, Lipsey, & Higgins, 2018). More importantly, our meta-analysis can be characterized as having a “divergent” framework (i.e., being focused on effect size variation and its explanation). For this review framework, it is recommended to include all relevant effect sizes instead of averaging on a sample level (López-López et al., 2018).

Therefore, we will use an analysis strategy that allows for the inclusion of multiple effect sizes from the same sample by modeling the arising statistical dependencies: Three-level meta-analysis (Van den Noortgate et al., 2013). Three-level meta-analyses can include an intermediate level to represent outcomes within studies. Specifically, classical meta-analysis has two levels (and sources of variation): level 1: participants (sampling error) and level 2: studies (difference in effect size between studies). Three-level meta-analysis with multiple effect sizes per sample has the following three levels (sources of variation): level 1: participants (sampling error), level 2: outcomes (difference in effect size between outcomes within studies), and level 3: studies (variation in effect size between studies). In contrast to traditional meta-analyses, three-level models result in two heterogeneity estimates, i.e., heterogeneity between outcomes within studies (level 2) and heterogeneity between studies (level 3).

Three-level-meta-analysis has been shown to perform well in the case of multiple effect sizes reported per sample (Van den Noortgate et al., 2013). Similarly, it has been shown to have favorable properties for moderator analyses (López-López et al., 2017). Specifically, three-level models have an acceptable type-1-error rate for level 3 (study-level) and level 2 (outcome-level) moderator variables if there is a moderate ( k > 20) and small ( k >10) number of studies, respectively. Regarding statistical power, the method consistently outperformed alternative (i.e., robust) meta-analysis methods (López-López et al., 2017).

All analyses will be conducted using the open source statistics software R (R Core Team, 2018). Three-level meta-analysis models will be implemented using the meta3 function from the metaSEM package (Cheung, 2015). The overall mean effect size and its 95%-confidence interval will be assessed as the intercept of a three-level model including all coded effect sizes with the sample as a clustering variable. The presence of heterogeneity will be determined using an overall Q-statistic for the homogeneity of effect sizes. It will be further interpreted utilizing the variance estimates (τ^2) and their statistical significance on level 2 (between outcomes within studies) and level 3 (between studies), respectively.

Moderator analyses will be performed by entering relevant moderators as predictors in the three-level model. Statistical significance of individual moderators will be assessed using a likelihood-ratio test by comparison to the same model without the respective moderator. This will be achieved using the anova function in R. The interpretation of moderator effects, if significant, will be based on their linear coefficients in the three-level model.

A more detailed analytic strategy, the concrete pre-planned models, and the alpha-level will also be provided in an addendum to this pre-registration before the coding of relevant studies.

Publication bias

Given the potential exaggeration of mean effect sizes by publication bias (Greenwald, 1975; Rothstein, Sutton, & Borenstein, 2006), meta-analyses should attempt to correct for it (see also Carter, Schönbrodt, Gervais, & Hilgard, 2017). One frequently method for this is trim and fill (Duval, Tweedie, 2000), but its performance has been shown to be inadequate (Moreno et al., 2009; Carter, Schönbrodt, Gervais, & Hilgard, 2017). More promising methods are the meta-regression PET-PEESE (Stanley, Doucouliagos, 2014) and especially the three-parameter selection model (Iyengar & Greenhouse, 1988; Vevea & Hedges, 1995). Carter et al. (2017) provide a simulation study on the effectiveness of these methods in the classical meta-analysis.

In the case of our meta-analysis, publication bias correction is complicated by the multi-level nature of our data. It will be assessed by entering publication status (published vs. unpublished) as a moderator variable in our three-level model. While some methods to simultaneously evaluate and correct for publication bias (e.g., PET-PEESE) could be integrated into a three-level model, others (e.g., the three-parameter selection model) would require independent effect sizes. The real analytic strategy to assess and correct for publication bias will be provided in an addendum to this pre-registration before data analysis.

Sensitivity analyses

The robustness of our results will be assessed by assessing the presence and if necessary after removing outliers (e.g., Viechtbauer & Cheung, 2010). The real analytic strategy to detect and remove outliers from the main-analyses and the moderator-analyses will be provided in an addendum to this pre-registration before data analysis.

Confidence in cumulative evidence

The strength of the increasing evidence for STIs will be decided based on the overall mean effect size before as well as after correction for publication bias and exclusion of outliers. Similarly, the strength of the evidence for specific moderator variables will be decided based on their robustness in the sensitivity analyses. The quality of studies will not be coded and analyzed since this is less straightforward in social psychology compared to, e.g., clinical trials (e.g., Berkman et al., 2014).


We can write a custom essay

According to Your Specific Requirements

Order an essay

You May Also Find These Documents Helpful

Peculiarities of various assignment types

The educational process is diverse and full of interesting writing tasks which help students develop their academic abilities. Different assignments types are created by professionals in order to enhance students’ level of analytical, critical and writing skills and to vary the learning process. As a student, you will encounter numerous tasks of diverse complexities throughout your student life. Sometimes, maybe, too complicated! They have different peculiarities, structural...

Making decisions in health and social care

Critically analyses the concepts, features, and importance of costs and accounting in making decisions in health and social care Cost accounting is a method used in accounting to capture a company’s or organisation’s production costs. It assesses the input costs of every step in production, fixed costs like depreciation of capital equipment. Cost accounting measures and records costs individually then compare the input results via...

Сhildren development

Physical development 7-12 years By the age of 7 a child enjoys things such as bike riding and rollerblading they are now able to tie and untie shoelaces without adult help, they are now starting to understand what rules are and are able to follow simple rules. At 8-12 years a child improves the physical skills that they have already developed and start to see...

Forex international trading market

Introduction Forex exchange is on the rise in Namibia; resulting in more people wanting to learn how to trade to try to increase their income so that they can enhance their standard of living. Forex Foreign exchange identifies the process of converting domestic currency into international banknotes at particular exchange rates (Bofah, 2017, para.1). As the number of foreigners in Namibia is increasing, more Namibians...

Aristotelian idea of God

This image produced in 1544 shows emerging's of the Judeo-Christians and Aristotelian's traditions. Aristotle was very interested in the idea of motion and said “The world is in a constant state of motion and change”. An example of how the world is changing is the growth of trees and plants. Aristotle believed in a prime mover, which is the being which creates change in the...

Get Access To The Full Essay
Materials Daily
100,000+ Subjects
2000+ Topics
Free Plagiarism
All Materials
are Cataloged Well

Sorry, but copying text is forbidden on this website. If you need this or any other sample, we can send it to you via email.

By clicking "SEND", you agree to our terms of service and privacy policy. We'll occasionally send you account related and promo emails.
Sorry, but only registered users have full access

How about getting this access

Become a member

Your Answer Is Very Helpful For Us
Thank You A Lot!


Emma Taylor


Hi there!
Would you like to get such a paper?
How about getting a customized one?

Couldn't Find What You Looking For?

Get access to our huge knowledge base which is continuously updated

Next Update Will Be About:
14 : 59 : 59
Become a Member