Keywords
community of practice
software accessibility
How to Cite
Abstract
Sensitivity analysis is now considered a standard practice in environmental modeling. Several open-source libraries, such as the Sensitivity Analysis Library (SALib), have been published in the recent past aimed at simplifying the application of sensitivity analyses. Still, there remain issues in software usability and accessibility, as well as a lack of guidance in the interpretation of sensitivity analysis results. This paper describes the changes made and planned to SALib to advance the ease with which modelers may conduct sensitivity analysis and interpret results. We further offer our perspectives from the past 7 years of maintaining SALib for the consideration of those aspiring to launch their own software for sensitivity analysis, develop methodology, or those otherwise interested in becoming involved in a project like SALib. These include the value of a community of practice to foster best practices for sensitivity analysis, the potential for collaboration across different software (for sensitivity analysis) platforms, and the need to specifically support the software development that underpins computational science.
References
Aznar-Siguan, G., & Bresch, D. N. (2019). CLIMADA v1: A global weather and climate risk assessment platform. Geoscientific Model Development, 12(7), 3085–3097. https://doi.org/10.5194/gmd-12-3085-2019
Baroni, G., & Francke, T. (2020). An effective strategy for combining variance- and distribution-based global sensitivity analysis. Environmental Modelling & Software, 134, 104851. https://doi.org/10.1016/j.envsoft.2020.104851
Borgonovo, E. (2007). A new uncertainty importance measure. Reliability Engineering & System Safety, 92(6), 771–784. https://doi.org/10.1016/j.ress.2006.04.015
Bresch, D. N., & Aznar-Siguan, G. (2021). CLIMADA v1.4.1: Towards a globally consistent adaptation options appraisal tool. Geoscientific Model Development, 14(1), 351–363. https://doi.org/10.5194/gmd-14-351-2021
Campolongo, F., Cariboni, J., & Saltelli, A. (2007). An effective screening design for sensitivity analysis of large models. Environmental Modelling & Software, 22(10), 1509–1518. https://doi.org/10.1016/j.envsoft.2006.10.004
Conda-Forge Community. (2015). The conda-forge Project: Community-based Software Distribution Built on the conda Package Format and Ecosystem. https://doi.org/10.5281/ZENODO.4774216
Cukier, R. I., Fortuin, C. M., Shuler, K. E., Petschek, A. G., & Schaibly, J. H. (1973). Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory. The Journal of Chemical Physics, 59(8), 3873–3878. https://doi.org/10.1063/1.1680571
Cuntz, M., & Mai, J. (2020). pyeee: Parameter screening using Morris’ method or its extension of Efficient/Sequential Elementary Effects (2.0) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.3897550
Cuntz, M., Mai, J., Zink, M., Thober, S., Kumar, R., Schäfer, D., Schrön, M., Craven, J., Rakovec, O., Spieler, D., Prykhodko, V., Dalmasso, G., Musuuza, J., Langenberg, B., Attinger, S., & Samaniego, L. (2015). Computationally inexpensive identification of noninformative model parameters by sequential screening. Water Resources Research, 51(8), 6417–6441. https://doi.org/10.1002/2015WR016907
Douglas-Smith, D., Iwanaga, T., Croke, B. F. W., & Jakeman, A. J. (2020). Certain trends in uncertainty and sensitivity analysis: An overview of software tools and techniques. Environmental Modelling & Software, 124, 104588. https://doi.org/10.1016/j.envsoft.2019.104588
Downey, A. (2017). Modeling and Simulation in Python. Green Tea Press. https://github.com/AllenDowney/ModSimPy (Original work published 2016)
Ferretti, F., Saltelli, A., & Tarantola, S. (2016). Trends in sensitivity analysis practice in the last decade. Science of The Total Environment, 568, 666–670. https://doi.org/10.1016/j.scitotenv.2016.02.133
Fonseca, J., Thomas, D., Mok, R., Mosteiro-Romero, M., Happle, G., Rogenhofer, L., Jack-Hawthorne, Fazel Khayatian, Zhongming Shi, Riegelbauer, E., Ong, B. L., Orenkiwi, H, T., Paulneitzel, Sulzer, M., Molony, R., Elesawy, A., JOSE ANTONIO BELLO ACOSTA, Bosova, A., … Strusoftsawen. (2021). architecture-building-systems/CityEnergyAnalyst: CityEnergyAnalyst v3.22.0 (v3.22.0) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.4646225
Foramitti, J. (2021). JoelForamitti/agentpy [Python]. https://github.com/JoelForamitti/agentpy (Original work published 2020)
Foster, I. (1995). Designing and building parallel programs: Concepts and tools for parallel software engineering. Addison-Wesley.
Hadjimichael, A., Gold, D., Hadka, D., & Reed, P. (2020). Rhodium: Python Library for Many-Objective Robust Decision Making and Exploratory Modeling. Journal of Open Research Software, 8(1), 12. https://doi.org/10.5334/jors.293
Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., & Wilson, G. (2009). How do scientists develop and use scientific software? 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, 1–8. https://doi.org/10.1109/SECSE.2009.5069155
Herlihy, M., & Shavit, N. (2012). The Art of Multiprocessor Programming. Elsevier Science. http://www.123library.org/book_details/?id=53644
Herman, J., & Usher, W. (2017, January 10). SALib: An open-source Python library for Sensitivity Analysis. The Journal of Open Source Software. https://doi.org/10.21105/joss.00097
Hermans, F. (2021). The Programmer’s Brain: What every programmer needs to know about cognition. Manning Publications.
Iwanaga, T. (2021). ConnectedSystems/SALib-impact: V0.5. Zenodo. https://doi.org/10.5281/zenodo.5523624
Iwanaga, T., Wang, H.-H., Hamilton, S. H., Grimm, V., Koralewski, T. E., Salado, A., Elsawah, S., Razavi, S., Yang, J., Glynn, P., Badham, J., Voinov, A., Chen, M., Grant, W. E., Peterson, T. R., Frank, K., Shenk, G., Barton, C. M., Jakeman, A. J., & Little, J. C. (2021). Socio-technical scales in socio-environmental modeling: Managing a system-of-systems modeling approach. Environmental Modelling & Software, 104885. https://doi.org/10.1016/j.envsoft.2020.104885
Kelly, D. F. (2007). A Software Chasm: Software Engineering and Scientific Computing. IEEE Software, 24(6), 120–119. https://doi.org/10.1109/MS.2007.155
Kwakkel, J. H. (2017). The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making. Environmental Modelling & Software, 96, 239–250. https://doi.org/10.1016/j.envsoft.2017.06.054
Li, G., Rabitz, H., Yelvington, P. E., Oluwole, O. O., Bacon, F., Kolb, C. E., & Schoendorf, J. (2010). Global Sensitivity Analysis for Systems with Independent and/or Correlated Inputs. The Journal of Physical Chemistry A, 114(19), 6022–6032. https://doi.org/10.1021/jp9096919
Little, J. C., Hester, E. T., Elsawah, S., Filz, G. M., Sandu, A., Carey, C. C., Iwanaga, T., & Jakeman, A. J. (2019). A tiered, system-of-systems modeling framework for resolving complex socio-environmental policy issues. Environmental Modelling & Software, 112, 82–94. https://doi.org/10.1016/j.envsoft.2018.11.011
Marelli, S., & Sudret, B. (2014). UQLab: A Framework for Uncertainty Quantification in Matlab. Vulnerability, Uncertainty, and Risk, 2554–2563. https://doi.org/10.1061/9780784413609.257
McKerns, M. M., Strand, L., Sullivan, T., Fang, A., & Aivazis, M. A. G. (2012). Building a Framework for Predictive Science. ArXiv:1202.1056 [Cs]. http://arxiv.org/abs/1202.1056
Morris, M. D. (1991). Factorial Sampling Plans for Preliminary Computational Experiments. Technometrics, 33(2), 161–174. https://doi.org/10.1080/00401706.1991.10484804
Niet, T., Shivakumar, A., Gardumi, F., Usher, W., Williams, E., & Howells, M. (2021). Developing a community of practice around an open source energy modelling tool. Energy Strategy Reviews, 35, 100650. https://doi.org/10.1016/j.esr.2021.100650
Niyazov, Y., Vogel, C., Price, R., Lund, B., Judd, D., Akil, A., Mortonson, M., Schwartzman, J., & Shron, M. (2016). Open Access Meets Discoverability: Citations to Articles Posted to Academia.edu. PLOS ONE, 11(2), e0148257. https://doi.org/10.1371/journal.pone.0148257
Noacco, V., Sarrazin, F., Pianosi, F., & Wagener, T. (2019). Matlab/R workflows to assess critical choices in Global Sensitivity Analysis using the SAFE toolbox. MethodsX, 6, 2258–2280. https://doi.org/10.1016/j.mex.2019.09.033
Paleari, L., & Confalonieri, R. (2016). Sensitivity analysis of a sensitivity analysis: We are likely overlooking the impact of distributional assumptions. Ecological Modelling, 340, 57–63. https://doi.org/10.1016/j.ecolmodel.2016.09.008
Pianosi, F., Beven, K., Freer, J., Hall, J. W., Rougier, J., Stephenson, D. B., & Wagener, T. (2016). Sensitivity analysis of environmental models: A systematic review with practical workflow. Environmental Modelling & Software, 79, 214–232. https://doi.org/10.1016/j.envsoft.2016.02.008
Pianosi, F., Sarrazin, F., & Wagener, T. (2020). How successfully is open-source research software adopted? Results and implications of surveying the users of a sensitivity analysis toolbox. Environmental Modelling & Software, 124, 104579. https://doi.org/10.1016/j.envsoft.2019.104579
Pianosi, F., & Wagener, T. (2015). A simple and efficient method for global sensitivity analysis based on cumulative distribution functions. Environmental Modelling & Software, 67, 1–11. https://doi.org/10.1016/j.envsoft.2015.01.004
Pianosi, F., & Wagener, T. (2018). Distribution-based sensitivity analysis from a generic input-output sample. Environmental Modelling & Software, 108, 197–207. https://doi.org/10.1016/j.envsoft.2018.07.019
Plischke, E. (2010). An effective algorithm for computing global sensitivity indices (EASI). Reliability Engineering & System Safety, 95(4), 354–360. https://doi.org/10.1016/j.ress.2009.11.005
Plischke, E., Borgonovo, E., & Smith, C. L. (2013). Global sensitivity measures from given data. European Journal of Operational Research, 226(3), 536–550. https://doi.org/10.1016/j.ejor.2012.11.047
Puy, A., Piano, S. L., Saltelli, A., & Levin, S. A. (2021). sensobol: An R package to compute variance-based sensitivity indices. ArXiv Preprint ArXiv:2101.10103. https://arxiv.org/abs/2101.10103
Rabitz, H., Aliş, Ö. F., Shorter, J., & Shim, K. (1999). Efficient input—Output model representations. Computer Physics Communications, 117(1), 11–20. https://doi.org/10.1016/S0010-4655(98)00152-0
Razavi, S., & Gupta, H. V. (2015). What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models. Water Resources Research, 51(5), 3070–3092. https://doi.org/10.1002/2014WR016527
Razavi, S., Jakeman, A. J., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Lo Piano, S., Iwanaga, T., Becker, W., Tarantola, S., Guillaume, J. H. A., Jakeman, J., Gupta, H., Melillo, N., Rabitti, G., Chabridon, V., Duan, Q., Sun, X., … Maier, H. R. (2021). The Future of Sensitivity Analysis: An Essential Discipline for Systems Modeling and Policy Support. Environmental Modelling & Software, 137, 104954. https://doi.org/10.1016/j.envsoft.2020.104954
Razavi, S., Sheikholeslami, R., Gupta, H. V., & Haghnegahdar, A. (2019). VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis. Environmental Modelling & Software, 112, 95–107. https://doi.org/10.1016/j.envsoft.2018.10.005
Ruano, M. V., Ribes, J., Seco, A., & Ferrer, J. (2012). An improved sampling strategy based on trajectory design for application of the Morris method to systems with many input factors. Environmental Modelling & Software, 37, 103–109. https://doi.org/10.1016/j.envsoft.2012.03.008
Saltelli, A. (2002). Making best use of model evaluations to compute sensitivity indices. Computer Physics Communications, 145, 280–297. https://doi.org/10.1016/S0010-4655(02)00280-1
Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., & Wu, Q. (2019). Why So Many Published Sensitivity Analyses Are False: A Systematic Review of Sensitivity Analysis Practices. Environmental Modelling & Software, 114, 29–39.
Saltelli, A., & Annoni, P. (2010). How to avoid a perfunctory sensitivity analysis. Environmental Modelling and Software, 25(12), 1508–1517. https://doi.org/10.1016/j.envsoft.2010.04.012
Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., & Tarantola, S. (2010). Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Computer Physics Communications, 181(2), 259–270. https://doi.org/10.1016/j.cpc.2009.09.018
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., & Tarantola, S. (2008). Global Sensitivity Analysis: The Primer. Wiley. https://dx.doi.org/10.1002/9780470725184
Saltelli, A., Tarantola, S., & Chan, K. P.-S. (1999). A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output. Technometrics, 41(1), 39–56. https://doi.org/10.1080/00401706.1999.10485594
Sheikholeslami, R., Gharari, S., Papalexiou, S. M., & Clark, M. P. (2021). VISCOUS: A Variance-Based Sensitivity Analysis Using Copulas for Efficient Identification of Dominant Hydrological Processes. Water Resources Research, 57(7), e2020WR028435. https://doi.org/10.1029/2020WR028435
Sobol′, I. M. (2001). Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and Computers in Simulation, 55(1–3), 271–280. https://doi.org/10.1016/S0378-4754(00)00270-6
Sobol’, I. M., & Kucherenko, S. (2010). Derivative based global sensitivity measures. Procedia - Social and Behavioral Sciences, 2(6), 7745–7746. https://doi.org/10.1016/j.sbspro.2010.05.208
Steiner, M., Bourinet, J.-M., & Lahmer, T. (2019). An adaptive sampling method for global sensitivity analysis based on least-squares support vector regression. Reliability Engineering & System Safety, 183, 323–340. https://doi.org/10.1016/j.ress.2018.11.015
Sweller, J. (1988). Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4
Tarantola, S., Gatelli, D., & Mara, T. A. (2006). Random balance designs for the estimation of first order global sensitivity indices. Reliability Engineering & System Safety, 91(6), 717–727. https://doi.org/10.1016/j.ress.2005.06.003
Teplitskiy, M., Lu, G., & Duede, E. (2017). Amplifying the impact of open access: Wikipedia and the diffusion of science. Journal of the Association for Information Science and Technology, 68(9), 2116–2127. https://doi.org/10.1002/asi.23687
Tissot, J.-Y., & Prieur, C. (2012). Bias correction for the estimation of sensitivity indices based on random balance designs. Reliability Engineering & System Safety, 107, 205–213. https://doi.org/10.1016/j.ress.2012.06.010
Wagener, T., & Pianosi, F. (2019). What has Global Sensitivity Analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling. Earth-Science Reviews, 194, 1–18. https://doi.org/10.1016/j.earscirev.2019.04.006
Wilson, G. (2006). Software Carpentry: Getting Scientists to Write Better Code by Making Them More Productive. Computing in Science Engineering, 8(6), 66–69. https://doi.org/10.1109/MCSE.2006.122
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2022 Takuya Iwanaga, Will Usher, Jon Herman