File Download
There are no files associated with this item.
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1029/2020WR029471
- Scopus: eid_2-s2.0-85115750900
- WOS: WOS:000703704400025
- Find via
Supplementary
- Citations:
- Appears in Collections:
Article: Reliability Assessment of Machine Learning Models in Hydrological Predictions Through Metamorphic Testing
Title | Reliability Assessment of Machine Learning Models in Hydrological Predictions Through Metamorphic Testing |
---|---|
Authors | |
Issue Date | 2021 |
Citation | Water Resources Research, 2021, v. 57, n. 9, article no. e2020WR029471 How to Cite? |
Abstract | The reliability of the machine learning model prediction for a given input can be assessed by comparing it against the actual output. However, in hydrological studies, machine learning models are often adopted to predict future or unknown events, where the actual outputs are unavailable. The prediction accuracy of a model, which measures its average performance across an observed data set, may not be relevant for a specific input. This study presents a method based on metamorphic testing (MT), adopted from software engineering, to assess the prediction reliability where the actual outputs are unknown. In this method, the predictions for a group of related inputs are considered consistent only if the input and output follow certain relations that are deduced from the properties of the system being modeled. For instance, the predicted runoff volume should increase in a rainfall-runoff model as the rainfall magnitude of an input increases. In this study, the MT-based method was applied to assess the predictions made by various machine learning models that were trained to predict the magnitude of flood events in Germany. Surprisingly, the prediction accuracy of a model and its ability to provide consistent predictions were found to be uncorrelated. This study further investigated the factors influencing the assessment result of a given input, such as its similarity to observed data. Overall, this research shows that MT is an effective and simple method for detecting inconsistent model predictions and is recommended when a model is employed to making predictions under new conditions. |
Persistent Identifier | http://hdl.handle.net/10722/335871 |
ISSN | 2023 Impact Factor: 4.6 2023 SCImago Journal Rankings: 1.574 |
ISI Accession Number ID |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Yang, Yang | - |
dc.contributor.author | Chui, Ting Fong May | - |
dc.date.accessioned | 2023-12-28T08:49:22Z | - |
dc.date.available | 2023-12-28T08:49:22Z | - |
dc.date.issued | 2021 | - |
dc.identifier.citation | Water Resources Research, 2021, v. 57, n. 9, article no. e2020WR029471 | - |
dc.identifier.issn | 0043-1397 | - |
dc.identifier.uri | http://hdl.handle.net/10722/335871 | - |
dc.description.abstract | The reliability of the machine learning model prediction for a given input can be assessed by comparing it against the actual output. However, in hydrological studies, machine learning models are often adopted to predict future or unknown events, where the actual outputs are unavailable. The prediction accuracy of a model, which measures its average performance across an observed data set, may not be relevant for a specific input. This study presents a method based on metamorphic testing (MT), adopted from software engineering, to assess the prediction reliability where the actual outputs are unknown. In this method, the predictions for a group of related inputs are considered consistent only if the input and output follow certain relations that are deduced from the properties of the system being modeled. For instance, the predicted runoff volume should increase in a rainfall-runoff model as the rainfall magnitude of an input increases. In this study, the MT-based method was applied to assess the predictions made by various machine learning models that were trained to predict the magnitude of flood events in Germany. Surprisingly, the prediction accuracy of a model and its ability to provide consistent predictions were found to be uncorrelated. This study further investigated the factors influencing the assessment result of a given input, such as its similarity to observed data. Overall, this research shows that MT is an effective and simple method for detecting inconsistent model predictions and is recommended when a model is employed to making predictions under new conditions. | - |
dc.language | eng | - |
dc.relation.ispartof | Water Resources Research | - |
dc.title | Reliability Assessment of Machine Learning Models in Hydrological Predictions Through Metamorphic Testing | - |
dc.type | Article | - |
dc.description.nature | link_to_subscribed_fulltext | - |
dc.identifier.doi | 10.1029/2020WR029471 | - |
dc.identifier.scopus | eid_2-s2.0-85115750900 | - |
dc.identifier.volume | 57 | - |
dc.identifier.issue | 9 | - |
dc.identifier.spage | article no. e2020WR029471 | - |
dc.identifier.epage | article no. e2020WR029471 | - |
dc.identifier.eissn | 1944-7973 | - |
dc.identifier.isi | WOS:000703704400025 | - |