Copyright and Licensing
Articles accepted for publication will be licensed under the Creative Commons BY-NC-SA. Authors must sign a non-exclusive distribution agreement after article acceptance.
Identifying Key Success Factors in Stopping Flaky Tests in Automated REST Service Testing
DOI:
https://doi.org/10.24215/16666038.18.e16Keywords:
Flaky tests, continuous integration, continuous deployment, continuous delivery, web service testingAbstract
A flaky test is a test which could fail or pass for the same version of a certain software code. In continuous software development environments, flaky tests represent a problem. It is difficult to get an effective and reliable testing pipeline with a set of flaky tests. Also, according to many practitioners, despite the persistence of flaky tests in software development, they have not drawn much attention from the research community. In this paper, we describe how a company faced this issue, and implemented solutions to solve flaky tests for REST web services. The paper concludes proposing a set of key success factors for stopping flaky tests in this type of testing.
Downloads
Download data is not yet available.
References
[1] M. Fowler, “Eradicating non-determinism in tests”, 2011. Available at: https://martinfowler.com/articles/nonDeterminism.html Accessed on 2018-01-27
[2] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests”, in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 643-653, 2014.
[3] F. Lacoste, “Killing the gatekeeper: Introducing a continuous integration system,” in Agile Conference ‘09, pp. 387-392, 2009.
[4] P. Sudarshan. “No more flaky tests on the Go team”. Available at: https://www.thoughtworks.com/insights/blog/no-more-flaky-testsgo-team Accessed on 2018-01-27
[5] TotT. “Avoiding flakey tests”. Available at: https://testing.googleblog.com/2008/04/tott-avoiding-flakeytests. html Accessed on 2018-01-27
[6] “Flakiness dashboard HOWTO”. Available at: http://www.chromium.org/developers/testing/flakiness-dashboard Accessed on 2018-01-27
[7] J. Micco, “Flaky Tests at Google and How We Mitigate Them”, 2016. Available at: https://testing.googleblog.com/2016/05/flaky-tests-at-googleand-how-we.html Accessed on 2018-01-27
[8] L. Hukkanen. Adopting Continuous Integration – A Case Study. M.Sc. thesis, Aalto University, 2015.
[9] E. Laukkanen, T. O. A. Lehtinen, J. Itkonen, M. Paasivaara, and C. Lassenius, “Bottom-up Adoption of Continuous Delivery in a StageGate Managed Software Organization,” in 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 1-10, 2016.
[10]F. Cannizzo, R. Clutton and R. Ramesh, "Pushing the boundaries of testing and continuous integration," in IEEE Agile Conference '08, pp. 501-505, 2008.
[11] J. Downs, J. Hoskins and B. Plimmer, “Status Communication in Agile Software Teams: A Case Study”, in Fifth International Conference on Software Engineering Advances, pp. 82-87, 2010.
[12]G. Gruver, M. Young, and P. Fulghum, A Practical Approach to LargeScale Agile Development: How HP Transformed LaserJet FutureSmart Firmware, New York, NY, USA: Addison-Wesley Professional, 1 ed., 2012
[13]S. Neely and S. Stolt, "Continuous delivery? easy! just change everything (well, maybe it is not that easy)," in IEEE Agile Conference '13, pp. 121-128, 2013.
[14] J. Süß and W. Billingsley, “Using continuous integration of code and content to teach software engineering with limited resources,” in Proceedings of the 34th International Conference on Software Engineering, pp. 1175-1184, 2012.
[15]A. Debbiche, M. Diener, and R. Berntsson Svensson, “Challenges When Adopting Continuous Integration: A Case Study,” Product-Focused Software Process Improvement, ser. Lecture Notes in Computer Science, vol. 8892, pp. 17–32, 2014.
[16]L. Elloussi, Determining flaky tests from test failures. Master Thesis, University of Illinois at Urbana-Champaign, 2015.
[17]A. Miller, “A hundred days of continuous integration,” in IEEE Agile Conference ‘08, pp. 289-293, 2008.
[18]Q. Luo, L. Eloussi, F. Hariri and D. Marinov, “Can We Trust Test Outcomes?”, 2014. Available at: https://pdfs.semanticscholar.org/a4b2/f4b9bcfdd0e83323570c40b893310f41e979.pdf Accessed on 2018-01-27
[19]M. Collin, Mastering Selenium WebDriver, Birmingham, UK, Packt Publishing, 2015.
[20]F. Palomba and A. Zaidman, “Does refactoring of test smells induce fixing flaky tests?” in Proceedings of the IEEE International Conference on Software Maintenance (ICSME), pp. 1-12, 2017.
[21] A. Gyori, Proactively detecting unreliable tests. PhD Thesis, University of Illinois at Urbana-Champaign, 2017.
[22] G. Brajnik, A. Baruzzo and S. Fabbro, “Modelbased continuous integration testing of responsiveness of web applications,” in Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1-2, 2015.
[23]E.G. Smith, “Automated Test Results Processing,” in Proceedings of the STAREAST 2001 Conference, pp. 1-13, 2001.
[24]M.A. Mascheroni, M.K. Cogliolo and E. Irrazábal. “Automatic detection of Web Incompatibilities using Digital Image Processing,” Electronic Journal of Informatics and Operations Research (SADIO EJS), Special Issue dedicated to JAIIO 2016, Vol. 16, No. 1, pp. 29-45, 2017.
[25]E. Dolstra, R. Vliegendhart and J. Pouwelse, "Crowdsourcing GUI tests," in Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation (ICST), pp. 332-341, 2013.
[26]B. Busjaeger and T. Xie, “Learning for test prioritization: an industrial case study,” in Proceedings of the 24th International Symposium on Foundations of Software Engineering, pp. 975–980, 2016.
[27]M. Eyl, C. Reichmann, and K. Müller-Glaser, “Fast feedback from automated tests executed with the product build,” in Proceedings of the International Conference on Software Quality, pp. 199–210, 2016.
[28]S. Elbaum, G. Rothermel, and J. Penix. “Techniques for improving regression testing in continuous integration development environments,” in Proceedings of the International Symposium on Foundations of Software Engineering, pp. 235–245, 2014.
[29]E. Laukkanen, J. Itkonen, and C. Lassenius. “Problems, Causes and Solutions When Adopting Continuous Delivery - A Systematic Literature Review,” Information and Software Technology, Vol. 8, pp. 55-79, 2016.
[30] J. Penix, “Large-scale test automation in the cloud”, in Proceedings of the 34th IEEE International Conference on Software Engineering (ICSE), page 1122, 2012.
[31]M. Gousset, A. Krishnamoorthy, B. Keller and S. Timm, Professional Application Lifecycle Management with Visual Studio 2010: with Team Foundation Server 2010, New Jersey, USA, Wiley Publishing, 2010.
[32]P. Duvall, S. Matyas and A. Glover, Continuous integration: improving software quality and reducing risk, Addison-Wesley, 2007.
[33] J. Humble and D. Farley, Continuous delivery: reliable software releases through build, test, and deployment automation, Boston, Addison-Wesley, 2011.
[34]M. Erder and P. Pureur, Continuous architecture: Sustainable architecture in an agile and cloud-centric world, Morgan Kaufmann, 2015.
[35]E. Givoni, N. Albert, Z. Ravitz, T.Q. Nguyen and T. Nguyen, "Automated software testing and validation system", 2006, U.S. Patent No US 7,093,238 B2.
[36] S. Khurshid, C.S. Păsăreanu and W. Visser, “Test input generation with Java PathFinder: then and now”, in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1-2, 2018.
[37] J. Wolf and S. Yoon, “Automated Testing for Continuous Delivery Pipelines” (industrial talk), in Pacific NW Software Quality Conference, 2016.
[2] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests”, in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 643-653, 2014.
[3] F. Lacoste, “Killing the gatekeeper: Introducing a continuous integration system,” in Agile Conference ‘09, pp. 387-392, 2009.
[4] P. Sudarshan. “No more flaky tests on the Go team”. Available at: https://www.thoughtworks.com/insights/blog/no-more-flaky-testsgo-team Accessed on 2018-01-27
[5] TotT. “Avoiding flakey tests”. Available at: https://testing.googleblog.com/2008/04/tott-avoiding-flakeytests. html Accessed on 2018-01-27
[6] “Flakiness dashboard HOWTO”. Available at: http://www.chromium.org/developers/testing/flakiness-dashboard Accessed on 2018-01-27
[7] J. Micco, “Flaky Tests at Google and How We Mitigate Them”, 2016. Available at: https://testing.googleblog.com/2016/05/flaky-tests-at-googleand-how-we.html Accessed on 2018-01-27
[8] L. Hukkanen. Adopting Continuous Integration – A Case Study. M.Sc. thesis, Aalto University, 2015.
[9] E. Laukkanen, T. O. A. Lehtinen, J. Itkonen, M. Paasivaara, and C. Lassenius, “Bottom-up Adoption of Continuous Delivery in a StageGate Managed Software Organization,” in 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 1-10, 2016.
[10]F. Cannizzo, R. Clutton and R. Ramesh, "Pushing the boundaries of testing and continuous integration," in IEEE Agile Conference '08, pp. 501-505, 2008.
[11] J. Downs, J. Hoskins and B. Plimmer, “Status Communication in Agile Software Teams: A Case Study”, in Fifth International Conference on Software Engineering Advances, pp. 82-87, 2010.
[12]G. Gruver, M. Young, and P. Fulghum, A Practical Approach to LargeScale Agile Development: How HP Transformed LaserJet FutureSmart Firmware, New York, NY, USA: Addison-Wesley Professional, 1 ed., 2012
[13]S. Neely and S. Stolt, "Continuous delivery? easy! just change everything (well, maybe it is not that easy)," in IEEE Agile Conference '13, pp. 121-128, 2013.
[14] J. Süß and W. Billingsley, “Using continuous integration of code and content to teach software engineering with limited resources,” in Proceedings of the 34th International Conference on Software Engineering, pp. 1175-1184, 2012.
[15]A. Debbiche, M. Diener, and R. Berntsson Svensson, “Challenges When Adopting Continuous Integration: A Case Study,” Product-Focused Software Process Improvement, ser. Lecture Notes in Computer Science, vol. 8892, pp. 17–32, 2014.
[16]L. Elloussi, Determining flaky tests from test failures. Master Thesis, University of Illinois at Urbana-Champaign, 2015.
[17]A. Miller, “A hundred days of continuous integration,” in IEEE Agile Conference ‘08, pp. 289-293, 2008.
[18]Q. Luo, L. Eloussi, F. Hariri and D. Marinov, “Can We Trust Test Outcomes?”, 2014. Available at: https://pdfs.semanticscholar.org/a4b2/f4b9bcfdd0e83323570c40b893310f41e979.pdf Accessed on 2018-01-27
[19]M. Collin, Mastering Selenium WebDriver, Birmingham, UK, Packt Publishing, 2015.
[20]F. Palomba and A. Zaidman, “Does refactoring of test smells induce fixing flaky tests?” in Proceedings of the IEEE International Conference on Software Maintenance (ICSME), pp. 1-12, 2017.
[21] A. Gyori, Proactively detecting unreliable tests. PhD Thesis, University of Illinois at Urbana-Champaign, 2017.
[22] G. Brajnik, A. Baruzzo and S. Fabbro, “Modelbased continuous integration testing of responsiveness of web applications,” in Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1-2, 2015.
[23]E.G. Smith, “Automated Test Results Processing,” in Proceedings of the STAREAST 2001 Conference, pp. 1-13, 2001.
[24]M.A. Mascheroni, M.K. Cogliolo and E. Irrazábal. “Automatic detection of Web Incompatibilities using Digital Image Processing,” Electronic Journal of Informatics and Operations Research (SADIO EJS), Special Issue dedicated to JAIIO 2016, Vol. 16, No. 1, pp. 29-45, 2017.
[25]E. Dolstra, R. Vliegendhart and J. Pouwelse, "Crowdsourcing GUI tests," in Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation (ICST), pp. 332-341, 2013.
[26]B. Busjaeger and T. Xie, “Learning for test prioritization: an industrial case study,” in Proceedings of the 24th International Symposium on Foundations of Software Engineering, pp. 975–980, 2016.
[27]M. Eyl, C. Reichmann, and K. Müller-Glaser, “Fast feedback from automated tests executed with the product build,” in Proceedings of the International Conference on Software Quality, pp. 199–210, 2016.
[28]S. Elbaum, G. Rothermel, and J. Penix. “Techniques for improving regression testing in continuous integration development environments,” in Proceedings of the International Symposium on Foundations of Software Engineering, pp. 235–245, 2014.
[29]E. Laukkanen, J. Itkonen, and C. Lassenius. “Problems, Causes and Solutions When Adopting Continuous Delivery - A Systematic Literature Review,” Information and Software Technology, Vol. 8, pp. 55-79, 2016.
[30] J. Penix, “Large-scale test automation in the cloud”, in Proceedings of the 34th IEEE International Conference on Software Engineering (ICSE), page 1122, 2012.
[31]M. Gousset, A. Krishnamoorthy, B. Keller and S. Timm, Professional Application Lifecycle Management with Visual Studio 2010: with Team Foundation Server 2010, New Jersey, USA, Wiley Publishing, 2010.
[32]P. Duvall, S. Matyas and A. Glover, Continuous integration: improving software quality and reducing risk, Addison-Wesley, 2007.
[33] J. Humble and D. Farley, Continuous delivery: reliable software releases through build, test, and deployment automation, Boston, Addison-Wesley, 2011.
[34]M. Erder and P. Pureur, Continuous architecture: Sustainable architecture in an agile and cloud-centric world, Morgan Kaufmann, 2015.
[35]E. Givoni, N. Albert, Z. Ravitz, T.Q. Nguyen and T. Nguyen, "Automated software testing and validation system", 2006, U.S. Patent No US 7,093,238 B2.
[36] S. Khurshid, C.S. Păsăreanu and W. Visser, “Test input generation with Java PathFinder: then and now”, in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1-2, 2018.
[37] J. Wolf and S. Yoon, “Automated Testing for Continuous Delivery Pipelines” (industrial talk), in Pacific NW Software Quality Conference, 2016.
Published
2018-10-09
Issue
Section
Original Articles
License
How to Cite
[1]
“Identifying Key Success Factors in Stopping Flaky Tests in Automated REST Service Testing”, JCS&T, vol. 18, no. 02, p. e16, Oct. 2018, doi: 10.24215/16666038.18.e16.
