Keywords
software testing
continuous integration
continuous development
How to Cite
Abstract
Software quality has been an issue for decades in many areas of scientific modelling for environment applications. Much of the software that has been developed is well-suited to supporting research investigation and application in one-off curated application environments, such as delivering solutions through community-based participatory approaches. However, when releasing the software into constrained production environments, with everyday operational challenges such as missing data, unintended user inputs, variable data quality, and values outside the bounds of those previously tested, lacks in software quality can become apparent. The lacks can result in lost time and effort to get the model running again, loss of trust in the model by users and, potentially, cessation in model use, and early and unexpected end-of-life. There have been many efforts through the years to encourage improved software quality, through approaches such as test-driven development, paired programming, more rigorous documentation, and better and broader user acceptance testing. The basis of many of these approaches is the art and science of software testing, through the whole development chain from unit testing, regression testing, integration testing, functional testing and to user testing. Continuous Integration/ Continuous Development (CI/CD)/ Continuous Deployment uses an automated or semi-automated process pipeline that promotes or progresses software through various stages of 'readiness' in different system realms, from development to testing to production. Faithful application of CI/CD promotes good quality software by requiring the software to pass a series of pre-determined tests before it is accepted into the next realm. Models must meet quality levels and pass automated tests to be able to be promoted. Additionally, governance gateways are used to check non-automated workflow components, such as manual testing. This paper explores the challenges and lessons learned in adoption and application of secure CI/CD in an operational environmental modelling enterprise, and suggests a minimum viable good practice approach for application to scientific environmental modelling.
References
Arvanitou, E. M., Ampatzoglou, A., Chatzigeorgiou, A., & Carver, J. C. (2021). Software engineering practices for scientific software development: A systematic mapping study. Journal of Systems and Software, 172. https://doi.org/10.1016/j.jss.2020.110848
Beck, K. (2002). Test Driven Development. By Example (Addison-Wesley Signature). Addison-Wesley Longman, Amsterdam.
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, Vol. 78, No. 1, pp. 1–3.
Dalal, S. R., Jain, A., Karunanithi, N., Leaton, J. M., Lott, C. M., Patton, G. C., & Horowitz, B. M. (1999). Model-based testing in practice. Proceedings of the 21st International Conference on Software Engineering, 285–294.
dos Santos, J., Martins, L. E. G., de Santiago Júnior, V. A., Povoa, L. V., & dos Santos, L. B. R. (2020). Software requirements testing approaches: a systematic literature review. Requirements Engineering, 25(3). https://doi.org/10.1007/s00766-019-00325-w
Dubois, P. F. (2012). Testing Scientific Programs. Computing in Science and Engg., 14(4), 69–73. https://doi.org/10.1109/MCSE.2012.84
Glowacki, T. J., Xiao, Y., & Steinle, P. (2012). Mesoscale Surface Analysis System for the Australian Domain: Design Issues, Development Status, and System Validation. Weather and Forecasting, 27(1), 141–157. https://doi.org/https://doi.org/10.1175/WAF-D-10-05063.1
Gupta, A., Poels, G., & Bera, P. (2022). Using Conceptual Models in Agile Software Development: A Possible Solution to Requirements Engineering Challenges in Agile Projects. IEEE Access, 10. https://doi.org/10.1109/ACCESS.2022.3221428
Heaton, D., & Carver, J. C. (2015). Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology, 67, 207–219. https://doi.org/https://doi.org/10.1016/j.infsof.2015.07.011
Iwanaga, T., Rahman, J., Partington, D., Croke, B., & Jakeman, A. J. (2018). Software Development Best Practices for Integrated Model Development. Proceedings International Conference on Environmental Modelling and Software, Fort Collins, Colorado https://doi.org/10.13140/RG.2.2.27034.34247
Jakeman, A. J., Elsawah, S., Wang, H.-H., Hamilton, S. H., Melsen, L., & Grimm, V. (2024). Towards normalizing good practice across the whole modeling cycle: its instrumentation and future research topics. Socio-Environmental Systems Modelling, 6, 18755.
Kanewala, U., & Bieman, J. M. (2014). Testing scientific software: A systematic literature review. In Information and Software Technology (Vol. 56, Issue 10). https://doi.org/10.1016/j.infsof.2014.05.006
Kanewala, U., & Chen, T. Y. (2019). Metamorphic Testing: A Simple Yet Effective Approach for Testing Scientific Software. Computing in Science & Engineering, 21(1), 66–72. https://doi.org/10.1109/MCSE.2018.2875368
Lucas, C., Potts, R. J., Zidikheri, M. J., Dare, R. A., Manickam, M., Wain, A., & Bear-Crozier, A. (2018). Improvements in the Detection and Prediction of Volcanic Ash for Aviation at the Australian Bureau of Meteorology. 98th American Meteorological Society Annual Meeting, 6pp.
Nanthaamornphong, A., & Carver, J. C. (2017). Test-Driven Development in scientific software: a survey. Software Quality Journal, 25(2). https://doi.org/10.1007/s11219-015-9292-4
Peng, Z., Lin, X., Simon, M., & Niu, N. (2021). Unit and regression tests of scientific software: A study on SWMM. Journal of Computational Science, 53. https://doi.org/10.1016/j.jocs.2021.101347
Pereira, N. S., Lima, P., Guerra, E., & Meirelles, P. (2022). Towards Automated Playtesting in Game Development. Proceedings of SBGames 2021. https://doi.org/10.5753/sbgames_estendido.2021.19666
Roberts, N., Ayliffe, B., Evans, G., Moseley, S., Rust, F., Sandford, C., Trzeciak, T., Abernethy, P., Beard, L., Crosswaite, N., Fitzpatrick, B., Flowerdew, J., Gale, T., Holly, L., Hopkinson, A., Hurst, K., Jackson, S., Jones, C., Mylne, K., … Worsfold, M. (2023). IMPROVER: The New Probabilistic Postprocessing System at the Met Office. Bulletin of the American Meteorological Society, 104, E680–E697. https://doi.org/10.1175/BAMS-D-21-0273.1
Shahin, M., Babar, M. A., & Zhu, L. (2017). Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. In IEEE Access (Vol. 5). https://doi.org/10.1109/ACCESS.2017.2685629
Staegemann, D., Volk, M., Pohl, M., Haertel, C., Hintsch, J., & Turowski, K. (2023). Identifying Guidelines for Test-Driven Development in Software Engineering—A Literature Review. Lecture Notes in Networks and Systems, 465. https://doi.org/10.1007/978-981-19-2397-5_30
Stein, A. F., Draxler, R. R., Rolph, G. D., Stunder, B. J. B., Cohen, M. D., & Ngan, F. (2015). NOAA’s HYSPLIT Atmospheric Transport and Dispersion Modeling System. Bulletin of the American Meteorological Society, 96(12), 2059–2077. https://doi.org/https://doi.org/10.1175/BAMS-D-14-00110.1
Taylor, A., & Brassington, G. B. (2017). Sea Level Forecasts Aggregated from Established Operational Systems. In Journal of Marine Science and Engineering (Vol. 5, Issue 3). https://doi.org/10.3390/jmse5030033
Uzun, B., & Tekinerdogan, B. (2018). Model-driven architecture based testing: A systematic literature review. Information and Software Technology, 102. https://doi.org/10.1016/j.infsof.2018.05.004
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2024 Dr Anja Schubert, Dr Robert Argent