Umbrella review: Methodological review of reviews published in peer-reviewed journals with a substantial focus on vocational education and training research

Abstract

Purpose: The growing public interest in vocational education and training (VET), most recently since the economic crisis of 2007/2008, has led to an exponential increase in articles with a vocational focus, underscoring the need for review studies for the purposes of systematic knowledge aggregation, clarification and interpretation. We assume that review studies follow the same minimum standards as other research methods: the review must be at least reproducible and thus the results verifiable or falsifiable. So far, however, the review methods used in VET research have not been investigated. Our purpose is to review the review procedures and methods used in published reviews of VET research to identify their current methodological quality. 

Approach: To classify the review studies, we initially developed a conceptual framework to distinguish different types of reviews. We then developed a methodological framework to assess the review methods used. Overall, to accelerate the review process, our review of reviews (or umbrella review) followed the rapid review approach: we limited our search to reviews in English published between 2014 and 2019 in peer-reviewed journals with a substantial VET focus and indexed in Scopus and/or Web of Science. Therefore, we did not examine all existing reviews in the field of VET research. Rather, our specific focus was on a core sector of scientific research: peer-reviewed articles in curated databases. Furthermore, we concentrated on the review procedures and methods used, not on the content of the reviews. 

Findings: We identified nine journals with a substantial VET focus, yielding a total of 1,283 published articles between 2014 and 2019, of which only 19 articles (1.48%) were literature reviews. Of these 19 reviews, six were excluded from our umbrella review because of unclear methodological procedures. Based on the review typology we developed, five of the remaining 13 reviews were conceptual in nature, four were scoping reviews, three were evidence-oriented, and one was critical in nature. None of the reviews examined focused on meta-syntheses, research methods or meta-analyses. In total, this resulted in current review gaps with respect to theory generation (meta-synthesis), practice of theory elaboration and testing (methodological review) and the determination of overall effects across single studies (meta-analysis). Finally, our examination of the reviews showed that their scope was mostly clearly presented. However, with regard to the process steps ‘data selection’ and ‘data processing’, only a few reviews fully met the requirements of the methodological framework.

Conclusion: Our review leads to four conclusions. 1) More systematic syntheses are needed because there is a substantial quantitative gap in review research. 2) In particular, there is a need for review studies with a focus on meta-synthesis, research methods and meta-analysis, as there is a current gap in these areas. 3) Reviews should be based on a review methodology with transparent and reproducible methods and verifiable or falsifiable results. The high number of subjective syntheses with unclear review procedures indicates that this mindset is not yet fully established in the field of VET research. 4) In the studies examined, there is a high degree of heterogeneity regarding to the accuracy and completeness of the methodological steps and data. The conceptual and methodological frameworks developed for the analysis can serve as guidelines for the conduct of reviews, and thus, the frameworks could contribute to the further development of the methodological basis of reviews.