Artificial intelligence, especially Long Short-Term Memory (LSTM) models, is widely used in hydrology to predict river runoff. While performance is usually measured using output-based metrics like Nash-Sutcliffe Efficiency (NSE) or Root Mean Square Error (RMSE), these don’t reveal how the model behaves internally. In our study, we explored this internal behavior to better understand when and why the model may fail.

We trained an LSTM-based surrogate of the National Water Model using NOAA precipitation data and extracted internal LSTM cell states which shows how the model thinks, during training, validation, and testing. We then used a method called convex hull analysis to identify whether test internal states were different from anything the model saw during training. We measured how far these states were from the training patterns to quantify novelty.

Our analysis showed that when test states were farther from the training hull, prediction errors increased and NSE values dropped. This helped us identify specific basins and dates where the model performed poorly.

By analyzing internal states, we created a new way to detect when and where the model is being pushed outside its comfort zone. This helps improve model trust, better generalization, and support safer decision-making in water management.