Toward SALib 2.0: Advancing the accessibility and interpretability of global sensitivity analyses
Article Full Text (PDF)
Model

Keywords

sensitivity analysis
community of practice
software accessibility

How to Cite

Iwanaga, T., Usher, W., & Herman, J. (2022). Toward SALib 2.0: Advancing the accessibility and interpretability of global sensitivity analyses. Socio-Environmental Systems Modelling, 4, 18155. https://doi.org/10.18174/sesmo.18155

Abstract

Sensitivity analysis is now considered a standard practice in environmental modeling. Several open-source libraries, such as the Sensitivity Analysis Library (SALib), have been published in the recent past aimed at simplifying the application of sensitivity analyses. Still, there remain issues in software usability and accessibility, as well as a lack of guidance in the interpretation of sensitivity analysis results. This paper describes the changes made and planned to SALib to advance the ease with which modelers may conduct sensitivity analysis and interpret results. We further offer our perspectives from the past 7 years of maintaining SALib for the consideration of those aspiring to launch their own software for sensitivity analysis, develop methodology, or those otherwise interested in becoming involved in a project like SALib. These include the value of a community of practice to foster best practices for sensitivity analysis, the potential for collaboration across different software (for sensitivity analysis) platforms, and the need to specifically support the software development that underpins computational science.

Article Full Text (PDF)
Model

References

Aznar-Siguan, G., & Bresch, D. N. (2019). CLIMADA v1: A global weather and climate risk assessment platform. Geoscientific Model Development, 12(7), 3085–3097. https://doi.org/10.5194/gmd-12-3085-2019

Baroni, G., & Francke, T. (2020). An effective strategy for combining variance- and distribution-based global sensitivity analysis. Environmental Modelling & Software, 134, 104851. https://doi.org/10.1016/j.envsoft.2020.104851

Borgonovo, E. (2007). A new uncertainty importance measure. Reliability Engineering & System Safety, 92(6), 771–784. https://doi.org/10.1016/j.ress.2006.04.015

Bresch, D. N., & Aznar-Siguan, G. (2021). CLIMADA v1.4.1: Towards a globally consistent adaptation options appraisal tool. Geoscientific Model Development, 14(1), 351–363. https://doi.org/10.5194/gmd-14-351-2021

Campolongo, F., Cariboni, J., & Saltelli, A. (2007). An effective screening design for sensitivity analysis of large models. Environmental Modelling & Software, 22(10), 1509–1518. https://doi.org/10.1016/j.envsoft.2006.10.004

Conda-Forge Community. (2015). The conda-forge Project: Community-based Software Distribution Built on the conda Package Format and Ecosystem. https://doi.org/10.5281/ZENODO.4774216

Cukier, R. I., Fortuin, C. M., Shuler, K. E., Petschek, A. G., & Schaibly, J. H. (1973). Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory. The Journal of Chemical Physics, 59(8), 3873–3878. https://doi.org/10.1063/1.1680571

Cuntz, M., & Mai, J. (2020). pyeee: Parameter screening using Morris’ method or its extension of Efficient/Sequential Elementary Effects (2.0) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.3897550

Cuntz, M., Mai, J., Zink, M., Thober, S., Kumar, R., Schäfer, D., Schrön, M., Craven, J., Rakovec, O., Spieler, D., Prykhodko, V., Dalmasso, G., Musuuza, J., Langenberg, B., Attinger, S., & Samaniego, L. (2015). Computationally inexpensive identification of noninformative model parameters by sequential screening. Water Resources Research, 51(8), 6417–6441. https://doi.org/10.1002/2015WR016907

Douglas-Smith, D., Iwanaga, T., Croke, B. F. W., & Jakeman, A. J. (2020). Certain trends in uncertainty and sensitivity analysis: An overview of software tools and techniques. Environmental Modelling & Software, 124, 104588. https://doi.org/10.1016/j.envsoft.2019.104588

Downey, A. (2017). Modeling and Simulation in Python. Green Tea Press. https://github.com/AllenDowney/ModSimPy (Original work published 2016)

Ferretti, F., Saltelli, A., & Tarantola, S. (2016). Trends in sensitivity analysis practice in the last decade. Science of The Total Environment, 568, 666–670. https://doi.org/10.1016/j.scitotenv.2016.02.133

Fonseca, J., Thomas, D., Mok, R., Mosteiro-Romero, M., Happle, G., Rogenhofer, L., Jack-Hawthorne, Fazel Khayatian, Zhongming Shi, Riegelbauer, E., Ong, B. L., Orenkiwi, H, T., Paulneitzel, Sulzer, M., Molony, R., Elesawy, A., JOSE ANTONIO BELLO ACOSTA, Bosova, A., … Strusoftsawen. (2021). architecture-building-systems/CityEnergyAnalyst: CityEnergyAnalyst v3.22.0 (v3.22.0) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.4646225

Foramitti, J. (2021). JoelForamitti/agentpy [Python]. https://github.com/JoelForamitti/agentpy (Original work published 2020)

Foster, I. (1995). Designing and building parallel programs: Concepts and tools for parallel software engineering. Addison-Wesley.

Hadjimichael, A., Gold, D., Hadka, D., & Reed, P. (2020). Rhodium: Python Library for Many-Objective Robust Decision Making and Exploratory Modeling. Journal of Open Research Software, 8(1), 12. https://doi.org/10.5334/jors.293

Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., & Wilson, G. (2009). How do scientists develop and use scientific software? 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, 1–8. https://doi.org/10.1109/SECSE.2009.5069155

Herlihy, M., & Shavit, N. (2012). The Art of Multiprocessor Programming. Elsevier Science. http://www.123library.org/book_details/?id=53644

Herman, J., & Usher, W. (2017, January 10). SALib: An open-source Python library for Sensitivity Analysis. The Journal of Open Source Software. https://doi.org/10.21105/joss.00097

Hermans, F. (2021). The Programmer’s Brain: What every programmer needs to know about cognition. Manning Publications.

Iwanaga, T. (2021). ConnectedSystems/SALib-impact: V0.5. Zenodo. https://doi.org/10.5281/zenodo.5523624

Iwanaga, T., Wang, H.-H., Hamilton, S. H., Grimm, V., Koralewski, T. E., Salado, A., Elsawah, S., Razavi, S., Yang, J., Glynn, P., Badham, J., Voinov, A., Chen, M., Grant, W. E., Peterson, T. R., Frank, K., Shenk, G., Barton, C. M., Jakeman, A. J., & Little, J. C. (2021). Socio-technical scales in socio-environmental modeling: Managing a system-of-systems modeling approach. Environmental Modelling & Software, 104885. https://doi.org/10.1016/j.envsoft.2020.104885

Kelly, D. F. (2007). A Software Chasm: Software Engineering and Scientific Computing. IEEE Software, 24(6), 120–119. https://doi.org/10.1109/MS.2007.155

Kwakkel, J. H. (2017). The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making. Environmental Modelling & Software, 96, 239–250. https://doi.org/10.1016/j.envsoft.2017.06.054

Li, G., Rabitz, H., Yelvington, P. E., Oluwole, O. O., Bacon, F., Kolb, C. E., & Schoendorf, J. (2010). Global Sensitivity Analysis for Systems with Independent and/or Correlated Inputs. The Journal of Physical Chemistry A, 114(19), 6022–6032. https://doi.org/10.1021/jp9096919

Little, J. C., Hester, E. T., Elsawah, S., Filz, G. M., Sandu, A., Carey, C. C., Iwanaga, T., & Jakeman, A. J. (2019). A tiered, system-of-systems modeling framework for resolving complex socio-environmental policy issues. Environmental Modelling & Software, 112, 82–94. https://doi.org/10.1016/j.envsoft.2018.11.011

Marelli, S., & Sudret, B. (2014). UQLab: A Framework for Uncertainty Quantification in Matlab. Vulnerability, Uncertainty, and Risk, 2554–2563. https://doi.org/10.1061/9780784413609.257

McKerns, M. M., Strand, L., Sullivan, T., Fang, A., & Aivazis, M. A. G. (2012). Building a Framework for Predictive Science. ArXiv:1202.1056 [Cs]. http://arxiv.org/abs/1202.1056

Morris, M. D. (1991). Factorial Sampling Plans for Preliminary Computational Experiments. Technometrics, 33(2), 161–174. https://doi.org/10.1080/00401706.1991.10484804

Niet, T., Shivakumar, A., Gardumi, F., Usher, W., Williams, E., & Howells, M. (2021). Developing a community of practice around an open source energy modelling tool. Energy Strategy Reviews, 35, 100650. https://doi.org/10.1016/j.esr.2021.100650

Niyazov, Y., Vogel, C., Price, R., Lund, B., Judd, D., Akil, A., Mortonson, M., Schwartzman, J., & Shron, M. (2016). Open Access Meets Discoverability: Citations to Articles Posted to Academia.edu. PLOS ONE, 11(2), e0148257. https://doi.org/10.1371/journal.pone.0148257

Noacco, V., Sarrazin, F., Pianosi, F., & Wagener, T. (2019). Matlab/R workflows to assess critical choices in Global Sensitivity Analysis using the SAFE toolbox. MethodsX, 6, 2258–2280. https://doi.org/10.1016/j.mex.2019.09.033

Paleari, L., & Confalonieri, R. (2016). Sensitivity analysis of a sensitivity analysis: We are likely overlooking the impact of distributional assumptions. Ecological Modelling, 340, 57–63. https://doi.org/10.1016/j.ecolmodel.2016.09.008

Pianosi, F., Beven, K., Freer, J., Hall, J. W., Rougier, J., Stephenson, D. B., & Wagener, T. (2016). Sensitivity analysis of environmental models: A systematic review with practical workflow. Environmental Modelling & Software, 79, 214–232. https://doi.org/10.1016/j.envsoft.2016.02.008

Pianosi, F., Sarrazin, F., & Wagener, T. (2020). How successfully is open-source research software adopted? Results and implications of surveying the users of a sensitivity analysis toolbox. Environmental Modelling & Software, 124, 104579. https://doi.org/10.1016/j.envsoft.2019.104579

Pianosi, F., & Wagener, T. (2015). A simple and efficient method for global sensitivity analysis based on cumulative distribution functions. Environmental Modelling & Software, 67, 1–11. https://doi.org/10.1016/j.envsoft.2015.01.004

Pianosi, F., & Wagener, T. (2018). Distribution-based sensitivity analysis from a generic input-output sample. Environmental Modelling & Software, 108, 197–207. https://doi.org/10.1016/j.envsoft.2018.07.019

Plischke, E. (2010). An effective algorithm for computing global sensitivity indices (EASI). Reliability Engineering & System Safety, 95(4), 354–360. https://doi.org/10.1016/j.ress.2009.11.005

Plischke, E., Borgonovo, E., & Smith, C. L. (2013). Global sensitivity measures from given data. European Journal of Operational Research, 226(3), 536–550. https://doi.org/10.1016/j.ejor.2012.11.047

Puy, A., Piano, S. L., Saltelli, A., & Levin, S. A. (2021). sensobol: An R package to compute variance-based sensitivity indices. ArXiv Preprint ArXiv:2101.10103. https://arxiv.org/abs/2101.10103

Rabitz, H., Aliş, Ö. F., Shorter, J., & Shim, K. (1999). Efficient input—Output model representations. Computer Physics Communications, 117(1), 11–20. https://doi.org/10.1016/S0010-4655(98)00152-0

Razavi, S., & Gupta, H. V. (2015). What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models. Water Resources Research, 51(5), 3070–3092. https://doi.org/10.1002/2014WR016527

Razavi, S., Jakeman, A. J., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Lo Piano, S., Iwanaga, T., Becker, W., Tarantola, S., Guillaume, J. H. A., Jakeman, J., Gupta, H., Melillo, N., Rabitti, G., Chabridon, V., Duan, Q., Sun, X., … Maier, H. R. (2021). The Future of Sensitivity Analysis: An Essential Discipline for Systems Modeling and Policy Support. Environmental Modelling & Software, 137, 104954. https://doi.org/10.1016/j.envsoft.2020.104954

Razavi, S., Sheikholeslami, R., Gupta, H. V., & Haghnegahdar, A. (2019). VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis. Environmental Modelling & Software, 112, 95–107. https://doi.org/10.1016/j.envsoft.2018.10.005

Ruano, M. V., Ribes, J., Seco, A., & Ferrer, J. (2012). An improved sampling strategy based on trajectory design for application of the Morris method to systems with many input factors. Environmental Modelling & Software, 37, 103–109. https://doi.org/10.1016/j.envsoft.2012.03.008

Saltelli, A. (2002). Making best use of model evaluations to compute sensitivity indices. Computer Physics Communications, 145, 280–297. https://doi.org/10.1016/S0010-4655(02)00280-1

Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., & Wu, Q. (2019). Why So Many Published Sensitivity Analyses Are False: A Systematic Review of Sensitivity Analysis Practices. Environmental Modelling & Software, 114, 29–39.

Saltelli, A., & Annoni, P. (2010). How to avoid a perfunctory sensitivity analysis. Environmental Modelling and Software, 25(12), 1508–1517. https://doi.org/10.1016/j.envsoft.2010.04.012

Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., & Tarantola, S. (2010). Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Computer Physics Communications, 181(2), 259–270. https://doi.org/10.1016/j.cpc.2009.09.018

Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., & Tarantola, S. (2008). Global Sensitivity Analysis: The Primer. Wiley. https://dx.doi.org/10.1002/9780470725184

Saltelli, A., Tarantola, S., & Chan, K. P.-S. (1999). A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output. Technometrics, 41(1), 39–56. https://doi.org/10.1080/00401706.1999.10485594

Sheikholeslami, R., Gharari, S., Papalexiou, S. M., & Clark, M. P. (2021). VISCOUS: A Variance-Based Sensitivity Analysis Using Copulas for Efficient Identification of Dominant Hydrological Processes. Water Resources Research, 57(7), e2020WR028435. https://doi.org/10.1029/2020WR028435

Sobol′, I. M. (2001). Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and Computers in Simulation, 55(1–3), 271–280. https://doi.org/10.1016/S0378-4754(00)00270-6

Sobol’, I. M., & Kucherenko, S. (2010). Derivative based global sensitivity measures. Procedia - Social and Behavioral Sciences, 2(6), 7745–7746. https://doi.org/10.1016/j.sbspro.2010.05.208

Steiner, M., Bourinet, J.-M., & Lahmer, T. (2019). An adaptive sampling method for global sensitivity analysis based on least-squares support vector regression. Reliability Engineering & System Safety, 183, 323–340. https://doi.org/10.1016/j.ress.2018.11.015

Sweller, J. (1988). Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4

Tarantola, S., Gatelli, D., & Mara, T. A. (2006). Random balance designs for the estimation of first order global sensitivity indices. Reliability Engineering & System Safety, 91(6), 717–727. https://doi.org/10.1016/j.ress.2005.06.003

Teplitskiy, M., Lu, G., & Duede, E. (2017). Amplifying the impact of open access: Wikipedia and the diffusion of science. Journal of the Association for Information Science and Technology, 68(9), 2116–2127. https://doi.org/10.1002/asi.23687

Tissot, J.-Y., & Prieur, C. (2012). Bias correction for the estimation of sensitivity indices based on random balance designs. Reliability Engineering & System Safety, 107, 205–213. https://doi.org/10.1016/j.ress.2012.06.010

Wagener, T., & Pianosi, F. (2019). What has Global Sensitivity Analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling. Earth-Science Reviews, 194, 1–18. https://doi.org/10.1016/j.earscirev.2019.04.006

Wilson, G. (2006). Software Carpentry: Getting Scientists to Write Better Code by Making Them More Productive. Computing in Science Engineering, 8(6), 66–69. https://doi.org/10.1109/MCSE.2006.122

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Copyright (c) 2022 Takuya Iwanaga, Will Usher, Jon Herman