Educational value of Systematic Mapping Studies: A replication study.

  • Jose L. Barros-Justo E.S.E.I., University of Vigo, Ourense, 32004, Spain
  • Nelson Martínez-Araujo E.S.E.I., University of Vigo, Ourense, 32004, Spain
  • David N. Olivieri E.S.E.I., University of Vigo, Ourense, 32004, Spain
  • Alejandro González-García E.S.E.I., University of Vigo, Ourense, 32004, Spain
Keywords: Educational value, Evidence Based Software engineering, Replication Studies, Systematic Mapping Studies


Contribution: The goal of this study is to validate the results of a prior study on the educational value of systematic mapping studies (SMS) and evaluate the possibility of generalizing those results to similar contexts. Background: since the inception of the Evidence Based Software Engineering (EBSE) paradigm in 2004, few studies have aimed at evaluating their impact in the academic field, and the educational values that SMS can offer to the students. In 2010, three renowned researchers presented an article on this subject at ICSE'10. The study made interesting contributions. We believe that, after 8 years, a replication of that study can be useful to assess the original findings and to guide future educational interventions. Research Questions: What educational value does a SMS provide? Methodology: a replication of a multi-case holistic study. Findings: This replication confirms all but two of the results from the original work. The differences may be due to slight variations in the context. The experience of undertaking an SMS provided the students with both research skills and a broad understanding of a research area. We encourage other researchers to replicate this study, to gain further insights into the educational value of this method.


Download data is not yet available.


B. A. Kitchenham, D. Budgen, and P. Brereton, Evidence-Based Software Engineering and Systematic Reviews, vol. 4. CRC Press, 2015.

B. Kitchenham and S. Charters, “Guidelines for performing Systematic Literature Reviews in Software Engineering,” Tech. report, EBSE Tech. Report. EBSE-2007-01, vol. 2, p. 1051, 2007.

K. Petersen, S. Vakkalanka, and L. Kuzniarz, “Guidelines for conducting systematic mapping studies in software engineering: An update,” Inf. Softw. Technol., vol. 64, pp. 1–18, 2015.

K. Petersen and C. Gencel, “Worldviews, research methods, and their relationship to validity in empirical software engineering research,” Softw. Meas. 2013 Eighth Int. Conf. Softw. Process Prod. Meas. (IWSM-

MENSURA), 2013 Jt. Conf. 23rd Int. Work., pp. 81–89, 2013.

B. Kitchenham, D. Budgen, and P. Brereton, “Using mapping studies as the basis for further research - A participant-observer case study,” Inf. Softw. Technol., no. 53, pp. 638–651, 2011.

B. Kitchenham and P. Brereton, “A systematic review of systematic review process research in software engineering,” Inf. Softw. Technol., vol. 55, no. 12, pp. 2049–2075, 2013.

K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, “Systematic Mapping Studies in Software Engineering,” 12th Int. Conf. Eval. Assess. Softw. Eng., vol. 17, pp. 1–10, 2007.

B. Kitchenham, P. Brereton, and D. Budgen, “The educational value of mapping studies of software engineering literature,” in 2010 ACM/IEEE 32nd International Conference on Software Engineering, 2010, vol. 1, pp. 589–598. 3, pp. 175–181, 2014.

C. Catal, “Teaching Evidence-Based Software Engineering to Master Students: A Single Lecture within a Course or an Entire Semester-Long Course?,” ACM SIGSOFT Softw. Eng. Notes, vol. 38, no. 2, pp. 1–2, 2013.

D. Castelluccia and G. Visaggio, “Teaching evidence-based software engineering:

learning by a collaborative mapping study of open source software,” ACM SIGSOFT Softw. Eng. Notes, vol. 38, pp. 1–4, 2013.

M. Lavallee, P. N. Robillard, and R. Mirsalari, “Performing systematic literature reviews with novices: An iterative approach,” IEEE Trans. Educ., vol. 57, no.3, pp. 175–181, 2014.

J. Biolchini, P. G. Mian, A. C. C. Natali, and G. H. Travassos, “Systematic review in software engineering,” Syst. Eng. Comput. Sci. Dep. COPPE/UFRJ, Tech. Rep. ES, vol. 679, no. 5, p. 45, 2005.

B. Pejcinovic, “Using Systematic Literature Reviews to Enhance Student Learning,” 2015 ASEE Annu. Conf. Expo., p. 26.1685.1-26.1685.13, 2015.

R. K. Yin, Case study research: Design and methods. Sage publications, 2013.

José L. Barros-Justo, N. Martínez-Araujo, and A. González-García, “Educational Value of Systematic Mapping Studies: A Replication. Complementary material.,” 2017. [Online]. Available: [Accessed: 08-Jan-2019].

How to Cite
Barros-JustoJ. L., Martínez-AraujoN., OlivieriD. N., & González-GarcíaA. (2019). Educational value of Systematic Mapping Studies: A replication study. Journal of Computer Science and Technology, 19(2), e12.
Original Articles