Keywords
convergence
good modeling practice
How to Cite
Abstract
In global sensitivity analysis (GSA) of a model, a proper convergence analysis of metrics is essential for ensuring a level of confidence or trustworthiness in sensitivity results obtained, yet is somewhat deficient in practice. The level of confidence in sensitivity measures, particularly in relation to their influence and support for decisions from scientific, social and policy perspectives, is heavily reliant on the convergence of GSA. We review the literature and summarize the available methods for monitoring and assessing convergence of sensitivity measures based on application purposes. The aim is to expose the various choices for convergence assessment and encourage further testing of available methods to clarify their level of robustness. Furthermore, the review identifies a pressing need for comparative studies on convergence assessment methods to establish a clear hierarchy of effectiveness and encourages the adoption of systematic approaches for enhanced robustness in sensitivity analysis.
References
Akomeah, E., Lindenschmidt, K. E., & Chapra, S. C. (2019). Comparison of aquatic ecosystem functioning between eutrophic and hypereutrophic cold-region river-lake systems. Ecological Modelling, 393, 25–36. https://doi.org/10.1016/j.ecolmodel.2018.12.004
Anstett-Collin, F., Goffart, J., Mara, T., & Denis-Vidal, L. (2015). Sensitivity analysis of complex models: Coping with dynamic and static inputs. Reliability Engineering and System Safety, 134, 268–275. https://doi.org/10.1016/j.ress.2014.08.010
Archer, G. E. B., Saltelli, A., & Sobol, I. M. (1997). Sensitivity measures, anova-like techniques and the use of bootstrap. Journal of Statistical Computation and Simulation, 58(2), 99–120. https://doi.org/10.1080/00949659708811825
Asensio-Sevilla, M. I., Santos-Martín, M. T., Álvarez-León, D., & Ferragut-Canals, L. (2020). Global sensitivity analysis of fuel-type-dependent input variables of a simplified physical fire spread model. Mathematics and Computers in Simulation, 172, 33–44. https://doi.org/10.1016/j.matcom.2020.01.001
Awad, M., Senga Kiesse, T., Assaghir, Z., & Ventura, A. (2019). Convergence of sensitivity analysis methods for evaluating combined influences of model inputs. Reliability Engineering and System Safety, 189, 109–122. https://doi.org/10.1016/j.ress.2019.03.050
Bajracharya, A., Awoye, H., Stadnyk, T., & Asadzadeh, M. (2020). Time variant sensitivity analysis of hydrological model parameters in a cold region using flow signatures. Water, 12(4), 961. https://doi.org/10.3390/W12040961
Baroni, G., Scheiffele, L. M., Schrön, M., Ingwersen, J., & Oswald, S. E. (2018). Uncertainty, sensitivity and improvements in soil moisture estimation with cosmic-ray neutron sensing. Journal of Hydrology, 564, 873–887. https://doi.org/10.1016/j.jhydrol.2018.07.053
Baroni, G., & Francke, T. (2020). An effective strategy for combining variance- and distribution-based global sensitivity analysis. Environmental Modelling and Software, 134, 104851. https://doi.org/10.1016/j.envsoft.2020.104851
Benedetti, L., Claeys, F., Nopens, I., & Vanrolleghem, P. A. (2011). Assessing the convergence of LHS Monte Carlo simulations of wastewater treatment models. Water Science and Technology, 63(10), 2219–2224. https://doi.org/10.2166/wst.2011.453
Berezowski, T., Nossent, J., Chormaåski, J., & Batelaan, O. (2015). Spatial sensitivity analysis of snow cover data in a distributed rainfall-runoff model. Hydrology and Earth System Sciences, 19(4), 1887–1904. https://doi.org/10.5194/hess-19-1887-2015
Borroni, C. G. (2013). A new rank correlation measure. Statistical Papers, 54(2), 255–270. https://doi.org/10.1007/s00362-011-0423-0
Branger, F., Giraudet, L. G., Guivarch, C., & Quirion, P. (2015). Global sensitivity analysis of an energy-economy model of the residential building sector. Environmental Modelling and Software, 70, 45–54. https://doi.org/10.1016/j.envsoft.2015.03.021
Brembilla, E., Mardaljevic, J., & Hopfe, C. J. (2015). Sensitivity analysis studying the impact of reflectance values assigned in climate-based daylight modelling. 14th International Conference of IBPSA - Building Simulation 2015, BS 2015, Conference Proceedings, September, 1197–1204.
Budamala, V., & Baburao Mahindrakar, A. (2020). Integration of Adaptive Emulators and Sensitivity Analysis for Enhancement of Complex Hydrological Models. Environmental Processes, 7(4), 1235–1253. https://doi.org/10.1007/s40710-020-00468-x
Campolongo, F., Cariboni, J., & Saltelli, A. (2007). An effective screening design for sensitivity analysis of large models. Environmental Modelling and Software, 22(10), 1509–1518. https://doi.org/10.1016/j.envsoft.2006.10.004
Castaings, W., Borgonovo, E., Morris, M. D., & Tarantola, S. (2012). Sampling strategies in density-based sensitivity analysis. Environmental Modelling and Software, 38, 13–26. https://doi.org/10.1016/j.envsoft.2012.04.017
Chaney, N. W., Herman, J. D., Ek, M. B., & Wood, E. F. (2016). Deriving global parameter estimates for the Noah land surface model using FLUXNET and machine learning. Journal of Geophysical Research, 121(22), 13,218-13,235. https://doi.org/10.1002/2016JD024821
Chisari, C., Rizzano, G., Amadio, C., & Galdi, V. (2018). Sensitivity analysis and calibration of phenomenological models for seismic analyses. Soil Dynamics and Earthquake Engineering, 109, 10–22. https://doi.org/10.1016/j.soildyn.2018.02.024
Ciric, C., Ciffroy, P., & Charles, S. (2012). Use of sensitivity analysis to identify influential and non-influential parameters within an aquatic ecosystem model. Ecological Modelling, 246(C), 119–130. https://doi.org/10.1016/j.ecolmodel.2012.06.024
Cloke, H. L., Pappenberger, F., & Renaud, J. P. (2008). Multi-Method Global Sensitivity Analysis (MMGSA) for modelling floodplain hydrological processes. Hydrological Processes, 22(11), 1660–1674. https://doi.org/10.1002/hyp.6734
Confalonieri, R., Bellocchi, G., Bregaglio, S., Donatelli, M., & Acutis, M. (2010a). Comparison of sensitivity analysis techniques: A case study with the rice model WARM. Ecological Modelling, 221(16), 1897–1906. https://doi.org/10.1016/j.ecolmodel.2010.04.021
Confalonieri, R., Bregaglio, S., & Acutis, M. (2012). Quantifying plasticity in simulation models. Ecological Modelling, 225, 159–166. https://doi.org/10.1016/j.ecolmodel.2011.11.022
Confalonieri, R., Bellocchi, G., Tarantola, S., Acutis, M., Donatelli, M., & Genovese, G. (2010b). Sensitivity analysis of the rice model WARM in Europe: Exploring the effects of different locations, climates and methods of analysis on model sensitivity to crop parameters. Environmental Modelling & Software, 25(4), 479–488. https://doi.org/10.1016/j.envsoft.2009.10.005
Coolen-Maturi, T. (2014). A new weighted rank coefficient of concordance. Journal of Applied Statistics, 41(8), 1721–1745. https://doi.org/10.1080/02664763.2014.889664
Cosenza, A., Mannina, G., Vanrolleghem, P. A., & Neumann, M. B. (2013). Global sensitivity analysis in wastewater applications: A comprehensive comparison of different methods. Environmental Modelling and Software, 49, 40–52. https://doi.org/10.1016/j.envsoft.2013.07.009
Cosenza, A., Mannina, G., Vanrolleghem, P. A., & Neumann, M. B. (2014). Variance-based sensitivity analysis for wastewater treatment plant modelling. Science of the Total Environment, 470–471, 1068–1077. https://doi.org/10.1016/j.scitotenv.2013.10.069
Crosetto, M., Tarantola, S., & Saltelli, A. (2000). Sensitivity and uncertainty analysis in spatial modelling based on GIS. Agriculture, Ecosystems and Environment, 81(1), 71–79. https://doi.org/10.1016/S0167-8809(00)00169-9
Cruz May, E., Bassam, A., Ricalde, L. J., Escalante Soberanis, M. A., Oubram, O., May Tzuc, O., Alanis, A. Y., & Livas-García, A. (2021). Global sensitivity analysis for a real-time electricity market forecast by a machine learning approach: A case study of Mexico. International Journal of Electrical Power and Energy Systems, 135, 107505. https://doi.org/10.1016/j.ijepes.2021.107505
Dai, H., & Ye, M. (2015). Variance-based global sensitivity analysis for multiple scenarios and models with implementation using sparse grid collocation. Journal of Hydrology, 528, 286–300. https://doi.org/10.1016/j.jhydrol.2015.06.034
Devak, M., & Dhanya, C. T. (2017). Sensitivity analysis of hydrological models: Review and way forward. Journal of Water and Climate Change, 8(4), 557–575. https://doi.org/10.2166/wcc.2017.149
Do, N. C., & Razavi, S. (2020). Correlation Effects? A Major but Often Neglected Component in Sensitivity and Uncertainty Analysis. Water Resources Research, 56(3). https://doi.org/10.1029/2019WR025436
Douglas-Smith, D., Iwanaga, T., Croke, B. F. W., & Jakeman, A. J. (2020). Certain trends in uncertainty and sensitivity analysis: An overview of software tools and techniques. Environmental Modelling and Software, 124, 104588. https://doi.org/10.1016/j.envsoft.2019.104588
Estrada, V., & Diaz, M. S. (2010). Global sensitivity analysis in the development of first principle-based eutrophication models. Environmental Modelling and Software, 25(12), 1539–1551. https://doi.org/10.1016/j.envsoft.2010.06.009
Faggianelli, G. A., Mora, L., & Merheb, R. (2017). Uncertainty quantification for Energy Savings Performance Contracting: Application to an office building. Energy and Buildings, 152, 61–72. https://doi.org/10.1016/j.enbuild.2017.07.022
Garcia, D., Arostegui, I., & Prellezo, R. (2019). Robust combination of the Morris and Sobol methods in complex multidimensional models. Environmental Modelling and Software, 122, 104517. https://doi.org/10.1016/j.envsoft.2019.104517
Ge, Q., & Menendez, M. (2014). An Efficient Sensitivity Analysis Approach for Computationally Expensive Microscopic Traffic Simulation Models. International Journal of Transportation, 2(2), 49–64. https://doi.org/10.14257/ijt.2014.2.2.04
Ghasemizade, M., Baroni, G., Abbaspour, K., & Schirmer, M. (2017). Combined analysis of time-varying sensitivity and identifiability indices to diagnose the response of a complex environmental model. Environmental Modelling and Software, 88, 22–34. https://doi.org/10.1016/j.envsoft.2016.10.011
Gilardelli, C., Confalonieri, R., Cappelli, G. A., & Bellocchi, G. (2018). Sensitivity of WOFOST-based modelling solutions to crop parameters under climate change. Ecological Modelling, 368, 1–14. https://doi.org/10.1016/j.ecolmodel.2017.11.003
Gilquin, L., Prieur, C., Arnaud, E., & Monod, H. (2021). Iterative estimation of Sobol’ indices based on replicated designs. Computational and Applied Mathematics, 40(1), 1–23. https://doi.org/10.1007/s40314-020-01402-5
Gokarakonda, S., van Treeck, C., & Rawal, R. (2019). Influence of building design and control parameters on the potential of mixed-mode buildings in India. Building and Environment, 148, 157–172. https://doi.org/10.1016/j.buildenv.2018.10.043
Gómez-Delgado, M., & Tarantola, S. (2006). GLOBAL sensitivity analysis, GIS and multi-criteria evaluation for a sustainable planning of a hazardous waste disposal site in Spain. International Journal of Geographical Information Science, 20(4), 449–466. https://doi.org/10.1080/13658810600607709
Gschwend, D., Soltic, P., Edinger, P., Wokaun, A., & Vogel, F. (2017). Performance evaluation of gasoline alternatives using a thermodynamic spark-ignition engine model. Sustainable Energy and Fuels, 1(9), 1991–2005. https://doi.org/10.1039/C7SE00276A
Guillaume, J. H. A., Jakeman, J. D., Marsili-Libelli, S., Asher, M., Brunner, P., Croke, B., Hill, M. C., Jakeman, A. J., Keesman, K. J., Razavi, S., & Stigter, J. D. (2019). Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose. Environmental Modelling and Software, 119, 418–432. https://doi.org/10.1016/j.envsoft.2019.07.007
Gupta, H. V., & Razavi, S. (2018). Revisiting the Basis of Sensitivity Analysis for Dynamical Earth System Models. Water Resources Research, 54(11), 8692–8717. https://doi.org/10.1029/2018WR022668
Hart, J., & Gremaud, P. (2019). Robustness of the Sobol’ indices to distributional uncertainty. International Journal for Uncertainty Quantification, 9(5), 453–469. https://doi.org/10.1615/Int.J.UncertaintyQuantification.2019030553
Hartmann, A., Šimůnek, J., Aidoo, M. K., Seidel, S. J., & Lazarovitch, N. (2018). Implementation and Application of a Root Growth Module in HYDRUS. Vadose Zone Journal, 17(1), 170040. https://doi.org/10.2136/vzj2017.02.0040
Helton, J. C., Davis, F. J., & Johnson, J. D. (2005). A comparison of uncertainty and sensitivity analysis results obtained with random and Latin hypercube sampling. Reliability Engineering and System Safety, 89(3), 305–330. https://doi.org/10.1016/j.ress.2004.09.006
Herman, J. D., Kollat, J. B., Reed, P. M., & Wagener, T. (2013). Technical Note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models. Hydrology and Earth System Sciences, 17(7), 2893–2903. https://doi.org/10.5194/hess-17-2893-2013
Homma, T., & Saltelli, A. (1996). Importance measures in global sensitivity analysis of nonlinear models. Reliability Engineering & System Safety, 52(1), 1–17. https://doi.org/10.1016/0951-8320(96)00002-6
Hornberger, G., & Spear, R. (1981). Approach to the preliminary analysis of environmental systems. J. Environ. Manage.; (United States), 12:1, 7–18. https://www.osti.gov/biblio/6396608
Hsieh, N. H., Bois, F. Y., Tsakalozou, E., Ni, Z., Yoon, M., Sun, W., Klein, M., Reisfeld, B., & Chiu, W. A. (2021). A Bayesian population physiologically based pharmacokinetic absorption modeling approach to support generic drug development: application to bupropion hydrochloride oral dosage forms. Journal of Pharmacokinetics and Pharmacodynamics, 5. https://doi.org/10.1007/s10928-021-09778-5
Hsieh, N. H., Reisfeld, B., Bois, F. Y., & Chiu, W. A. (2018). Applying a global sensitivity analysis workflow to improve the computational efficiencies in physiologically-based pharmacokinetic modeling. Frontiers in Pharmacology, 9, 1–17. https://doi.org/10.3389/fphar.2018.00588
Hsieh, N. H., Reisfeld, B., & Chiu, W. A. (2020). pksensi: An R package to apply global sensitivity analysis in physiologically based kinetic modeling. SoftwareX, 12, 100609. https://doi.org/10.1016/j.softx.2020.100609
Hu, Y., Garcia-Cabrejo, O., Cai, X., Valocchi, A. J., & DuPont, B. (2015). Global sensitivity analysis for large-scale socio-hydrological models using Hadoop. Environmental Modelling & Software, 73, 231–243. https://doi.org/10.1016/j.envsoft.2015.08.015
Huo, X., Gupta, H., Niu, G. Y., Gong, W., & Duan, Q. (2019). Parameter Sensitivity Analysis for Computationally Intensive Spatially Distributed Dynamical Environmental Systems Models. Journal of Advances in Modeling Earth Systems, 11(9), 2896–2909. https://doi.org/10.1029/2018MS001573
Iman, R. L., & Conover, W. J. (1987). A Measure of Top-Down Correlation. Technometrics, 29(3), 351. https://doi.org/10.2307/1269344
Isaksson, A., Wallman, M., Göransson, H., & Gustafsson, M. G. (2008). Cross-validation and bootstrapping are unreliable in small sample classification. Pattern Recognition Letters, 29(14), 1960–1965. https://doi.org/10.1016/j.patrec.2008.06.018
Jabloun, M., Li, X., Zhang, X., Tao, F., Hu, C., & Olesen, J. E. (2018). Sensitivity of simulated crop yield and nitrate leaching of the wheat-maize cropping system in the North China Plain to model parameters. Agricultural and Forest Meteorology, 263, 25–40. https://doi.org/10.1016/j.agrformet.2018.08.002
Jakeman, A. J., Letcher, R. A., & Norton, J. P. (2006). Ten iterative steps in development and evaluation of environmental models. Environmental Modelling and Software, 21(5), 602–614. https://doi.org/10.1016/j.envsoft.2006.01.004
Jakeman, J. (2023). PyApprox: A software package for sensitivity analysis, Bayesian inference, optimal experimental design, and multi-fidelity uncertainty quantification and surrogate modeling. Environmental Modelling and Software, 170, 105825. https://doi.org/10.1016/j.envsoft.2023.105825
Janssen, H. (2013). Monte-Carlo based uncertainty analysis: Sampling efficiency and sampling convergence. Reliability Engineering and System Safety, 109, 123–132. https://doi.org/10.1016/j.ress.2012.08.003
Kavetski, D., & Clark, M. P. (2010). Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction. Water Resources Research, 46(10), 2009WR008896. https://doi.org/10.1029/2009WR008896
KC, U., Aryal, J., Garg, S., & Hilton, J. (2021). Global sensitivity analysis for uncertainty quantification in fire spread models. Environmental Modelling and Software, 143, 105110. https://doi.org/10.1016/j.envsoft.2021.105110
KC, U., Garg, S., Hilton, J., & Aryal, J. (2020). A cloud-based framework for sensitivity analysis of natural hazard models. Environmental Modelling and Software, 134, 104800. https://doi.org/10.1016/j.envsoft.2020.104800
Kelleher, C., & Wagener, T. (2011). Ten guidelines for effective data visualization in scientific publications. Environmental Modelling & Software, 26(6), 822–827. https://doi.org/10.1016/j.envsoft.2010.12.006
Khan, S., & Kaklis, P. (2021). From regional sensitivity to intra-sensitivity for parametric analysis of free-form shapes: Application to ship design. Advanced Engineering Informatics, 49, 101314. https://doi.org/10.1016/j.aei.2021.101314
Khorashadi Zadeh, F., Nossent, J., Sarrazin, F., Pianosi, F., van Griensven, A., Wagener, T., & Bauwens, W. (2017). Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the SWAT model. Environmental Modelling and Software, 91, 210–222. https://doi.org/10.1016/j.envsoft.2017.02.001
Koo, H., Iwanaga, T., Croke, B. F. W., Jakeman, A. J., Yang, J., Wang, H. H., Sun, X., Lü, G., Li, X., Yue, T., Yuan, W., Liu, X., & Chen, M. (2020). Position paper: Sensitivity analysis of spatially distributed environmental models- a pragmatic framework for the exploration of uncertainty sources. Environmental Modelling and Software, 134, 104857. https://doi.org/10.1016/j.envsoft.2020.104857
Krishnan, P., & Aggarwal, P. (2018). Global sensitivity and uncertainty analyses of a web based crop simulation model (web InfoCrop wheat) for soil parameters. Plant and Soil, 423(1–2), 443–463. https://doi.org/10.1007/s11104-017-3498-0
Krishnan, P., Maity, P. P., & Kundu, M. (2021). Sensitivity analysis of cultivar parameters to simulate wheat crop growth and yield under moisture and temperature stress conditions. Heliyon, 7(7), e07602. https://doi.org/10.1016/j.heliyon.2021.e07602
Kroll, S., Dirckx, G., Donckels, B. M. R., Van Dorpe, M., Weemaes, M., & Willems, P. (2016). Modelling real-time control of WWTP influent flow under data scarcity. Water Science and Technology, 73(7), 1637–1643. https://doi.org/10.2166/wst.2015.641
Lebedeva, G., Sorokin, A., Faratian, D., Mullen, P., Goltsov, A., Langdon, S. P., Harrison, D. J., & Goryanin, I. (2012). Model-based global sensitivity analysis as applied to identification of anti-cancer drug targets and biomarkers of drug resistance in the ErbB2/3 network. European Journal of Pharmaceutical Sciences, 46(4), 244–258. https://doi.org/10.1016/j.ejps.2011.10.026
Leolini, L., Bregaglio, S., Moriondo, M., Ramos, M. C., Bindi, M., & Ginaldi, F. (2018). A model library to simulate grapevine growth and development: software implementation, sensitivity analysis and field level application. European Journal of Agronomy, 99, 92–105. https://doi.org/10.1016/j.eja.2018.06.006
Li, P., & Ren, L. (2019). Evaluating the effects of limited irrigation on crop water productivity and reducing deep groundwater exploitation in the North China Plain using an agro-hydrological model: I. Parameter sensitivity analysis, calibration and model validation. Journal of Hydrology, 574(2), 497–516. https://doi.org/10.1016/j.jhydrol.2019.04.053
Likhachev, D. V. (2019). Parametric sensitivity analysis as an essential ingredient of spectroscopic ellipsometry data modeling: An application of the Morris screening method. Journal of Applied Physics, 126(18). https://doi.org/10.1063/1.5126074
Liu, J., Liu, Z., Zhu, A. X., Shen, F., Lei, Q., & Duan, Z. (2019a). Global sensitivity analysis of the APSIM-Oryza rice growth model under different environmental conditions. Science of the Total Environment, 651, 953–968. https://doi.org/10.1016/j.scitotenv.2018.09.254
Liu, K., Xin, X., Ma, J., Zhang, J., & Yu, Q. (2019b). Sensitivity analysis of ship traffic in restricted two-way waterways considering the impact of LNG carriers. Ocean Engineering, 192, 106556. https://doi.org/10.1016/j.oceaneng.2019.106556
Locatelli, T., Tarantola, S., Gardiner, B., & Patenaude, G. (2017). Variance-based sensitivity analysis of a wind risk model - Model behaviour and lessons for forest modelling. Environmental Modelling and Software, 87, 84–109. https://doi.org/10.1016/j.envsoft.2016.10.010
Mai, J., & Tolson, B. A. (2019). Model Variable Augmentation (MVA) for Diagnostic Assessment of Sensitivity Analysis Results. Water Resources Research, 55(4), 2631–2651. https://doi.org/10.1029/2018WR023382
Mailier, J., Delmotte, A., Cloutier, M., Jolicoeur, M., & VandeWouwer, A. (2011). Sensitivity analysis for parameter identification of a plant growth model. In IFAC Proceedings Volumes (IFAC-PapersOnline) (Vol. 44, Issue 1 PART 1). IFAC. https://doi.org/10.3182/20110828-6-IT-1002.00315
Marino, S., Hogue, I. B., Ray, C. J., & Kirschner, D. E. (2008). A methodology for performing global uncertainty and sensitivity analysis in systems biology. Journal of Theoretical Biology, 254(1), 178–196. https://doi.org/10.1016/j.jtbi.2008.04.011
Matthews, C. J., Newton, D. B., Braddock, R. D., & Yu, B. (2007). Analysing the sensitivity behaviour of two hydrology models. Environmental Modeling and Assessment, 12(1), 27–41. https://doi.org/10.1007/s10666-006-9049-3
Medina, Y., & Muñoz, E. (2020a). A simple time-varying sensitivity analysis (TVSA) for assessment of temporal variability of hydrological processes. Water, 12(9), 2463. https://doi.org/10.3390/w12092463
Medina, Y., & Muñoz, E. (2020b). Analysis of the relative importance of model parameters in watersheds with different hydrological regimes. Water, 12(9), 2376. https://doi.org/10.3390/W12092376
Moreau, P., Viaud, V., Parnaudeau, V., Salmon-Monviola, J., & Durand, P. (2013). An approach for global sensitivity analysis of a complex environmental model to spatial inputs and parameters: A case study of an agro-hydrological model. Environmental Modelling and Software, 47, 74–87. https://doi.org/10.1016/j.envsoft.2013.04.006
Neumann, M. B. (2012). Comparison of sensitivity analysis methods for pollutant degradation modelling: A case study from drinking water treatment. Science of the Total Environment, 433, 530–537. https://doi.org/10.1016/j.scitotenv.2012.06.026
Nguyen, A. T., & Reiter, S. (2015). A performance comparison of sensitivity analysis methods for building energy models. Building Simulation, 8(6), 651–664. https://doi.org/10.1007/s12273-015-0245-4
Nossent, J., Elsen, P., & Bauwens, W. (2011). Sobol’ sensitivity analysis of a complex environmental model. Environmental Modelling and Software, 26(12), 1515–1525. https://doi.org/10.1016/j.envsoft.2011.08.010
Page, T., Smith, P., Beven, K., Pianosi, F., Sarrazin, F., Almeida, S., Holcombe, L., Freer, J., Chappell, N., & Wagener, T. (2023). Technical note: The CREDIBLE Uncertainty Estimation (CURE) toolbox: facilitating the communication of epistemic uncertainty. Hydrology and Earth System Sciences, 27(13), 2523–2534. https://doi.org/10.5194/hess-27-2523-2023
Paleari, L., & Confalonieri, R. (2016). Sensitivity analysis of a sensitivity analysis: We are likely overlooking the impact of distributional assumptions. Ecological Modelling. https://doi.org/10.1016/j.ecolmodel.2016.09.008
Paleari, L., Movedi, E., Zoli, M., Burato, A., Cecconi, I., Errahouly, J., Pecollo, E., Sorvillo, C., & Confalonieri, R. (2021). Sensitivity analysis using Morris: Just screening or an effective ranking method? Ecological Modelling, 455, 109648. https://doi.org/10.1016/j.ecolmodel.2021.109648
Pappenberger, F., Beven, K. J., Ratto, M., & Matgen, P. (2008). Multi-method global sensitivity analysis of flood inundation models. Advances in Water Resources, 31(1), 1–14. https://doi.org/10.1016/j.advwatres.2007.04.009
Peeters, L. J. M., Pagendam, D. E., Crosbie, R. S., Rachakonda, P. K., Dawes, W. R., Gao, L., Marvanek, S. P., Zhang, Y. Q., & McVicar, T. R. (2018). Determining the initial spatial extent of an environmental impact assessment with a probabilistic screening methodology. Environmental Modelling and Software, 109, 353–367. https://doi.org/10.1016/j.envsoft.2018.08.020
Peng, X., Adamowski, J., Inam, A., Alizadeh, M. R., & Albano, R. (2020). Development of a behaviour-pattern based global sensitivity analysis procedure for coupled socioeconomic and environmental models. Journal of Hydrology, 585, 124745. https://doi.org/10.1016/j.jhydrol.2020.124745
Pianosi, F., Beven, K., Freer, J., Hall, J. W., Rougier, J., Stephenson, D. B., & Wagener, T. (2016). Sensitivity analysis of environmental models: A systematic review with practical workflow. Environmental Modelling and Software, 79, 214–232. https://doi.org/10.1016/j.envsoft.2016.02.008
Pianosi, F., Sarrazin, F., & Wagener, T. (2015). A Matlab toolbox for Global Sensitivity Analysis. Environmental Modelling & Software, 70, 80–85. https://doi.org/10.1016/j.envsoft.2015.04.009
Pianosi, F., Sarrazin, F., & Wagener, T. (2020). How successfully is open-source research software adopted? Results and implications of surveying the users of a sensitivity analysis toolbox. Environmental Modelling and Software, 124. https://doi.org/10.1016/j.envsoft.2019.104579
Qian, G., & Mahdi, A. (2020). Sensitivity analysis methods in the biomedical sciences. Mathematical Biosciences, 323, 108306. https://doi.org/10.1016/j.mbs.2020.108306
Ravalico, J. K., Dandy, G. C., & Maier, H. R. (2010). Management Option Rank Equivalence (MORE) - A new method of sensitivity analysis for decision-making. Environmental Modelling and Software, 25(2), 171–181. https://doi.org/10.1016/j.envsoft.2009.06.012
Ravasi, R. A., Paleari, L., Vesely, F. M., Movedi, E., Thoelke, W., & Confalonieri, R. (2020). Ideotype definition to adapt legumes to climate change: A case study for field pea in Northern Italy. Agricultural and Forest Meteorology, 291. https://doi.org/10.1016/j.agrformet.2020.108081
Razavi, S., & Gupta, H. V. (2016a). A new framework for comprehensive, robust, and efficient global sensitivity analysis: 1. Theory. Water Resources Research, 52(1), 423–439. https://doi.org/10.1002/2015WR017558
Razavi, S., & Gupta, H. V. (2016b). A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application. Water Resources Research, 52(1), 440–455. https://doi.org/10.1002/2015WR017559
Razavi, S., & Gupta, H. V. (2019). A multi-method Generalized Global Sensitivity Matrix approach to accounting for the dynamical nature of earth and environmental systems models. Environmental Modelling and Software, 114, 1–11. https://doi.org/10.1016/j.envsoft.2018.12.002
Razavi, S., Jakeman, A., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Lo Piano, S., Iwanaga, T., Becker, W., Tarantola, S., Guillaume, J. H. A., Jakeman, J., Gupta, H., Melillo, N., Rabitti, G., Chabridon, V., Duan, Q., Sun, X., … Maier, H. R. (2021). The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support. Environmental Modelling and Software, 137. https://doi.org/10.1016/j.envsoft.2020.104954
Razavi, S., Sheikholeslami, R., Gupta, H. V., & Haghnegahdar, A. (2019). VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis. Environmental Modelling and Software, 112, 95–107. https://doi.org/10.1016/j.envsoft.2018.10.005
Reinhart, B. D., Frankenberger, J. R., Hay, C. H., Bowling, L. C., & Hancock, B. G. (2020). Development and Sensitivity Analysis of an Online Tool for Evaluating Drainage Water Recycling Decisions. Transactions of the ASABE, 63(6), 1991–2002. https://doi.org/10.13031/trans.13900
Reusser, D. E., Buytaert, W., & Zehe, E. (2011). Temporal dynamics of model parameter sensitivity for computationally expensive models with the Fourier amplitude sensitivity test. Water Resources Research, 47(7). https://doi.org/10.1029/2010WR009947
Robles, A., Ruano, M. V., Ribes, J., Seco, A., & Ferrer, J. (2014a). Global sensitivity analysis of a filtration model for submerged anaerobic membrane bioreactors (AnMBR). Bioresource Technology, 158, 365–373. https://doi.org/10.1016/j.biortech.2014.02.087
Robles, A., Ruano, M. V., Ribes, J., Seco, A., & Ferrer, J. (2014b). Model-based automatic tuning of a filtration control system for submerged anaerobic membrane bioreactors (AnMBR). Journal of Membrane Science, 465, 14–26. https://doi.org/10.1016/j.memsci.2014.04.012
Ruano, M. V., Ribes, J., Seco, A., & Ferrer, J. (2012). An improved sampling strategy based on trajectory design for application of the Morris method to systems with many input factors. Environmental Modelling and Software, 37, 103–109. https://doi.org/10.1016/j.envsoft.2012.03.008
Şalap-Ayça, S., Jankowski, P., Clarke, K. C., & Nara, A. (2021). Is less more? Experimenting with visual stacking of coincident maps for spatial global sensitivity analysis in urban land-use change modeling. Environmental Modelling and Software, 145. https://doi.org/10.1016/j.envsoft.2021.105181
Saltelli, A., Chan, K., & Scott, E. (2000). Sensitivity Analysis: Wiley Series in Probability and Statistics. Wiley.
Saltelli, Andrea, & Annoni, P. (2010). How to avoid a perfunctory sensitivity analysis. Environmental Modelling and Software, 25(12), 1508–1517. https://doi.org/10.1016/j.envsoft.2010.04.012
Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., & Tarantola, S. (2010). Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Computer Physics Communications, 181(2), 259–270. https://doi.org/10.1016/j.cpc.2009.09.018
Saltelli, A., & Bolado, R. (1998). An alternative way to compute Fourier amplitude sensitivity test (FAST). Computational Statistics and Data Analysis, 26(4), 445–460. https://doi.org/10.1016/S0167-9473(97)00043-1
Saltelli, A., Jakeman, A., Razavi, S., & Wu, Q. (2021). Sensitivity analysis: A discipline coming of age. Environmental Modelling & Software, 146, 105226. https://doi.org/10.1016/j.envsoft.2021.105226
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., & Tarantola, S. (2007). Global Sensitivity Analysis. The Primer. In International Statistical Review. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470725184
Saltelli, A., Tarantola, S., Campolongo, F., & Ratto, M. (2004). Sensitivity analysis in practice: a guide to assessing scientific models.
Salviano, L. O., Gasparin, E. E., Mattos, V. C. N., Barbizan, B., Saltara, F., de Mello, P. E. B., Dezan, D. J., & Yanagihara, J. I. (2021). Sensitivity analysis and optimization of a CO2 centrifugal compressor impeller with a vaneless diffuser. Structural and Multidisciplinary Optimization, 64(3), 1607–1627. https://doi.org/10.1007/s00158-021-02914-2
Sarrazin, F., Hartmann, A., Pianosi, F., Rosolem, R., & Wagener, T. (2018). V2Karst V1.1: A parsimonious large-scale integrated vegetation-recharge model to simulate the impact of climate and land cover change in karst regions. In Geoscientific Model Development (Vol. 11, Issue 12). https://doi.org/10.5194/gmd-11-4933-2018
Sarrazin, F., Pianosi, F., & Wagener, T. (2016). Global Sensitivity Analysis of environmental models: Convergence and validation. Environmental Modelling and Software, 79, 135–152. https://doi.org/10.1016/j.envsoft.2016.02.005
Savage, I. R. (1956). Contributions to the Theory of Rank Order Statistics-the Two-Sample Case. The Annals of Mathematical Statistics, 27(3), 590–615. https://www.jstor.org/stable/2237370
Seo, S. B., Duchnowski, E. M., O’Neal, M., Motta, A. T., Passelaigue, F., Kang, S., Pastore, G., Manera, A., Petrov, V., Huang, P. H., & Brown, N. R. (2021). Sensitivity analysis of BISON model for characterization of impact of experimental parameters on hydrogen migration and redistribution in zirconium-based alloys. Journal of Nuclear Materials, 550, 152941. https://doi.org/10.1016/j.jnucmat.2021.152941
Sheikholeslami, R., Gharari, S., Papalexiou, S. M., & Clark, M. P. (2021). VISCOUS: A Variance-Based Sensitivity Analysis Using Copulas for Efficient Identification of Dominant Hydrological Processes. Water Resources Research, 57(7), 1–24. https://doi.org/10.1029/2020WR028435
Sheikholeslami, R., & Razavi, S. (2017). Progressive Latin Hypercube Sampling: An efficient approach for robust sampling-based analysis of environmental models. Environmental Modelling and Software, 93, 109–126. https://doi.org/10.1016/j.envsoft.2017.03.010
Sheikholeslami, R., Razavi, S., Gupta, H. V., Becker, W., & Haghnegahdar, A. (2019a). Global sensitivity analysis for high-dimensional problems: How to objectively group factors and measure robustness and convergence while reducing computational cost. Environmental Modelling and Software, 111, 282–299. https://doi.org/10.1016/j.envsoft.2018.09.002
Sheikholeslami, R., Razavi, S., & Haghnegahdar, A. (2019b). What should we do when a model crashes? Recommendations for global sensitivity analysis of Earth and environmental systems models. Geoscientific Model Development, 12(10), 4275–4296. https://doi.org/10.5194/gmd-12-4275-2019
Sheikholeslami, R., Yassin, F., Lindenschmidt, K. E., & Razavi, S. (2017). Improved understanding of river ice processes using global sensitivity analysis approaches. Journal of Hydrologic Engineering, 22(11), 04017048. https://doi.org/10.1061/(ASCE)HE.1943-5584.0001574
Shin, M. J., Guillaume, J. H. A., Croke, B. F. W., & Jakeman, A. J. (2013). Addressing ten questions about conceptual rainfall-runoff models with global sensitivity analyses in R. Journal of Hydrology, 503, 135–152. https://doi.org/10.1016/j.jhydrol.2013.08.047
Shin, M. J., Guillaume, J. H. A., Croke, B. F. W., & Jakeman, A. J. (2015). A review of foundational methods for checking the structural identifiability of models: Results for rainfall-runoff. Journal of Hydrology, 520, 1–16. https://doi.org/10.1016/j.jhydrol.2014.11.040
Sieber, A., & Uhlenbrook, S. (2005). Sensitivity analyses of a distributed catchment model to verify the model structure. Journal of Hydrology, 310(1–4), 216–235. https://doi.org/10.1016/j.jhydrol.2005.01.004
Silvestro, P. C., Pignatti, S., Yang, H., Yang, G., Pascucci, S., Castaldi, F., & Casa, R. (2017). Sensitivity analysis of the Aquacrop and SAFYE crop models for the assessment of water limited winter wheat yield in regional scale applications. PLoS ONE, 12(11), 1–30. https://doi.org/10.1371/journal.pone.0187485
Sobol, M. (1993). Sensitivity Estimates for Nonlinear Mathematical Models. Mathematical Modeling and Computational Experiment, 1(4), 407–414. https://doi.org/1061-7590/93/04407-008$9.00
Song, X. M., Kong, F. Z., Zhan, C. S., Han, J. W., & Zhang, X. H. (2013a). Parameter identification and global sensitivity analysis of Xin’anjiang model using meta-modeling approach. Water Science and Engineering, 6(1), 1–17. https://doi.org/10.3882/j.issn.1674-2370.2013.01.001
Song, X., Bryan, B. A., Almeida, A. C., Paul, K. I., Zhao, G., & Ren, Y. (2013b). Time-dependent sensitivity of a process-based ecological model. Ecological Modelling, 265, 114–123. https://doi.org/10.1016/j.ecolmodel.2013.06.013
Song, X., Bryan, B. A., Paul, K. I., & Zhao, G. (2012). Variance-based sensitivity analysis of a forest growth model. Ecological Modelling, 247, 135–143. https://doi.org/10.1016/j.ecolmodel.2012.08.005
Song, X., Zhang, J., Zhan, C., Xuan, Y., Ye, M., & Xu, C. (2015). Global sensitivity analysis in hydrological modeling: Review of concepts, methods, theoretical framework, and applications. Journal of Hydrology, 523(225), 739–757. https://doi.org/10.1016/j.jhydrol.2015.02.013
Specka, X., Nendel, C., & Wieland, R. (2019). Temporal sensitivity analysis of the MONICA model: Application of two global approaches to analyze the dynamics of parameter sensitivity. Agriculture (Switzerland), 9(2), 37. https://doi.org/10.3390/agriculture9020037
Sreedevi, S., Eldho, T. I., Madhusoodhanan, C. G., & Jayasankar, T. (2019). Multiobjective sensitivity analysis and model parameterization approach for coupled streamflow and groundwater table depth simulations using SHETRAN in a wet humid tropical catchment. Journal of Hydrology, 579, 124217. https://doi.org/10.1016/j.jhydrol.2019.124217
Sudheer, K. P., Lakshmi, G., & Chaubey, I. (2011). Application of a pseudo simulator to evaluate the sensitivity of parameters in complex watershed models. Environmental Modelling and Software, 26(2), 135–143. https://doi.org/10.1016/j.envsoft.2010.07.007
Sun, X. (2021). SAConvergenceAnalysis. GitHub. https://github.com/Xifus/SAConvergenceAnalysis
Sun, X., Croke, B., Jakeman, A., & Roberts, S. (2022). Benchmarking Active Subspace methods of global sensitivity analysis against variance-based Sobol’ and Morris methods with established test functions. Environmental Modelling & Software, 149, 105310. https://doi.org/10.1016/j.envsoft.2022.105310
Sun, X., Croke, B., Roberts, S., & Jakeman, A. (2021). Comparing methods of randomizing Sobol′ sequences for improving uncertainty of metrics in variance-based global sensitivity estimation. Reliability Engineering & System Safety, 210, 107499. https://doi.org/10.1016/j.ress.2021.107499
Tan, J., Cui, Y., & Luo, Y. (2017). Assessment of uncertainty and sensitivity analyses for ORYZA model under different ranges of parameter variation. European Journal of Agronomy, 91, 54–62. https://doi.org/10.1016/j.eja.2017.09.001
Tang, Y., Reed, P., Wagener, T., & Van Werkhoven, K. (2007). Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation. Hydrology and Earth System Sciences, 11(2), 793–817. https://doi.org/10.5194/hess-11-793-2007
Tarantola, S., Becker, W., & Zeitz, D. (2012). A comparison of two sampling methods for global sensitivity analysis. Computer Physics Communications, 183(5), 1061–1072. https://doi.org/10.1016/j.cpc.2011.12.015
Tarantola, S., Ferretti, F., Lo Piano, S., Kozlova, M., Lachi, A., Rosati, R., Puy, A., Roy, P., Vannucci, G., Kuc-Czarnecka, M., & Saltelli, A. (2024). An annotated timeline of sensitivity analysis. Environmental Modelling & Software, 174, 105977. https://doi.org/10.1016/j.envsoft.2024.105977
Touhami, H. B., Lardy, R., Barra, V., & Bellocchi, G. (2013). Screening parameters in the Pasture Simulation model using the Morris method. Ecological Modelling, 266(1), 42–57. https://doi.org/10.1016/j.ecolmodel.2013.07.005
Touzani, S., & Busby, D. (2014). Screening Method Using the Derivative-based Global Sensitivity Indices with Application to Reservoir Simulator. Oil & Gas Science and Technology – Revue d’IFP Energies Nouvelles, 69(4), 619–632. https://doi.org/10.2516/ogst/2013195
Uliana, E. M., da Silva, D. D., Moreira, M. C., & Pereira, D. dos R. (2019). Global sensitivity analysis methods applied to hydrologic modeling with the SAC-SMA model. Engenharia Agricola, 39(1), 65–74. https://doi.org/10.1590/1809-4430-Eng.Agric.v39n1p65-74/2019
Upreti, D., Pignatti, S., Pascucci, S., Tolomio, M., Li, Z., Huang, W., & Casa, R. (2020). A Comparison of Moment-Independent and Variance-Based Global Sensitivity Analysis Approaches for Wheat Yield Estimation with the Aquacrop-OS Model. Agronomy, 10(4), 607. https://doi.org/10.3390/agronomy10040607
Van Werkhoven, K., Wagener, T., Reed, P., & Tang, Y. (2008). Characterization of watershed model behavior across a hydroclimatic gradient. Water Resources Research, 44(1), 1–16. https://doi.org/10.1029/2007WR006271
Vanrolleghem, P. A., Mannina, G., Cosenza, A., & Neumann, M. B. (2015). Global sensitivity analysis for urban water quality modelling: Terminology, convergence and comparison of different methods. Journal of Hydrology, 522, 339–352. https://doi.org/10.1016/j.jhydrol.2014.12.056
Vigna, S. (2015). A weighted correlation index for rankings with ties. WWW 2015 - Proceedings of the 24th International Conference on World Wide Web, 1166–1176. https://doi.org/10.1145/2736277.2741088
Wagener, T., & Pianosi, F. (2019). What has Global Sensitivity Analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling. Earth-Science Reviews, 194, 1–18. https://doi.org/10.1016/j.earscirev.2019.04.006
Wagener, T., Van Werkhoven, K., Reed, P., & Tang, Y. (2009). Multiobjective sensitivity analysis to understand the information content in streamflow observations for distributed watershed modeling. Water Resources Research, 45(2), 1–5. https://doi.org/10.1029/2008WR007347
Wang, A., & Solomatine, D. P. (2019). Practical experience of sensitivity analysis: Comparing six methods, on three hydrological models, with three performance criteria. Water, 11(5), 1062, 1–26. https://doi.org/10.3390/w11051062
Wang, C., Duan, Q., Gong, W., Ye, A., Di, Z., & Miao, C. (2014). An evaluation of adaptive surrogate modeling based optimization with two benchmark problems. Environmental Modelling and Software, 60, 167–179. https://doi.org/10.1016/j.envsoft.2014.05.026
Wang, H., Gong, W., Duan, Q., & Di, Z. (2020). Evaluation of parameter interaction effect of hydrological models using the sparse polynomial chaos (SPC) method. Environmental Modelling and Software, 125. https://doi.org/10.1016/j.envsoft.2019.104612
Wang, J., Li, X., Lu, L., & Fang, F. (2013). Parameter sensitivity analysis of crop growth models based on the extended Fourier Amplitude Sensitivity Test method. Environmental Modelling and Software, 48, 171–182. https://doi.org/10.1016/j.envsoft.2013.06.007
Wang, Q., Guillaume, J. H. A., Jakeman, J. D., Yang, T., Iwanaga, T., Croke, B., & Jakeman, A. J. (2022). Assessing the predictive impact of factor fixing with an adaptive uncertainty-based approach. Environmental Modelling and Software, 148, 105290. https://doi.org/10.1016/j.envsoft.2021.105290
Wang, S., Flipo, N., & Romary, T. (2018). Time-dependent global sensitivity analysis of the C-RIVE biogeochemical model in contrasted hydrological and trophic contexts. Water Research, 144, 341–355. https://doi.org/10.1016/j.watres.2018.07.033
Wate, P., Iglesias, M., Coors, V., & Robinson, D. (2020). Framework for emulation and uncertainty quantification of a stochastic building performance simulator. Applied Energy, 258, 113759. https://doi.org/10.1016/j.apenergy.2019.113759
Wu, Z., Ma, B., Wang, H., Hu, C., Lv, H., & Zhang, X. (2021). Identification of Sensitive Parameters of Urban Flood Model Based on Artificial Neural Network. Water Resources Management, 35(7), 2115–2128. https://doi.org/10.1007/s11269-021-02825-3
Yang, J. (2011). Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis. Environmental Modelling and Software, 26(4), 444–457. https://doi.org/10.1016/j.envsoft.2010.10.007
Yang, J., Jakeman, A., Fang, G., & Chen, X. (2018). Uncertainty analysis of a semi-distributed hydrologic model based on a Gaussian Process emulator. Environmental Modelling and Software, 101, 289–300. https://doi.org/10.1016/j.envsoft.2017.11.037
Yang, J., Liu, Y., Yang, W., & Chen, Y. (2012). Multi-Objective Sensitivity Analysis of a Fully Distributed Hydrologic Model WetSpa. Water Resources Management, 26(1), 109–128. https://doi.org/10.1007/s11269-011-9908-9
Zhan, C., Song, X., Xia, J., & Tong, C. (2013). An efficient integrated approach for global sensitivity analysis of hydrological model parameters. Environmental Modelling and Software, 41, 39–52. https://doi.org/10.1016/j.envsoft.2012.10.009
Zhan, Y., & Zhang, M. (2013). Application of a combined sensitivity analysis approach on a pesticide environmental risk indicator. Environmental Modelling and Software, 49, 129–140. https://doi.org/10.1016/j.envsoft.2013.08.005
Zhang, C., Chu, J., & Fu, G. (2013). Sobol’’s sensitivity analysis for a distributed hydrological model of Yichun River Basin, China. Journal of Hydrology, 480, 58–68. https://doi.org/10.1016/j.jhydrol.2012.12.005
Zhao, G., Bryan, B. A., & Song, X. (2014). Sensitivity and uncertainty analysis of the APSIM-wheat model: Interactions between cultivar, environmental, and management parameters. Ecological Modelling, 279, 1–11. https://doi.org/10.1016/j.ecolmodel.2014.02.003
Zhao, Y., Nan, Z., Yu, W., & Zhang, L. (2019). Calibrating a hydrological model by stratifying frozen ground types and seasons in a cold alpine basin. Water, 11(5), 985. https://doi.org/10.3390/w11050985
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2024 Xifu Sun, Anthony J. Jakeman, Barry F.W. Croke, Stephen G. Roberts, John D. Jakeman