A Hybrid Physics-Informed Deep Learning Framework for Robust Multivariate System Modelling Under Uncertainty

Authors

  • Sonu Pandey Independent Researcher, Financial Systems & AI, USA Author

DOI:

https://doi.org/10.67887/PD2601202005

Keywords:

Physics-informed learning, scientific machine learning, transformer architecture, uncertainty quantification, Robust dynamical systems, multivariate time-series modelling, distribution shift, hybrid deep learning

Abstract

Introduction: This study presents a unified physics-regularised Transformer framework for robust multivariate dynamical-system modelling under uncertainty. Conventional data-driven deep learning models can capture nonlinear temporal patterns but may produce physically inconsistent predictions under distribution shifts. In contrast, classical Physics-Informed Neural Networks (PINNs) improve physical consistency but often suffer from fixed loss weighting, gradient imbalance, and limited uncertainty calibration.

Methodology: To address these limitations, this study integrates three components into a single modelling framework: a gradient-norm-based adaptive physics-weighting strategy, an attention-based temporal encoder, and a heteroscedastic uncertainty-estimation layer. The governing equations are used as soft inductive biases rather than exact system descriptors, allowing the model to balance empirical data fitting with physics-guided regularisation.

Results & Discussion: Experiments on a synthetic multivariate dynamical-system dataset show consistent improvements over the LSTM, Pure Transformer, Standard PINN, and Deep Ensemble baselines in predictive accuracy, uncertainty calibration, and robustness to perturbations. The proposed model achieved lower RMSE, MAE, and negative log-likelihood than the implemented baselines while maintaining greater stability under noisy and missing-data conditions. Ablation results further indicate that adaptive weighting, physics regularisation, and heteroscedastic uncertainty modelling each contribute to the final performance.

Conclusion: The findings suggest that physics-regularised attention models can provide a useful direction for uncertainty-aware scientific machine learning. However, further validation on real-world benchmark datasets is required before broader claims of deployment can be made.

References

Quarteroni A, Gervasio P, Regazzoni F. Combining physics-based and data-driven models: advancing the frontiers of research with scientific machine learning. arXiv Preprint. 2025;arXiv:2501.18708.

https://doi.org/10.48550/arXiv.2501.18708.

Rudin C, Chen C, Chen Z, Huang H, Semenova L, Zhong C. Interpretable machine learning: fundamental principles and 10 grand challenges. Stat Surv. 2022; 16: 1-85. https://doi.org/10.1214/21-SS133.

Meng C, Griesemer S, Cao D, Seo S, Liu Y. When physics meets machine learning: a survey of physics-informed machine learning. Mach Learn Comput Sci Eng. 2025; 1(1): 20. https://doi.org/10.1007/s44379-025-00016-0.

Chen J, Shi Y. Stochastic model predictive control framework for resilient cyber-physical systems: review and perspectives. Philos Trans R Soc A Math Phys Eng Sci. 2021; 379(2207): 20200371. https://doi.org/10.1098/rsta.2020.0371.

Chen N. Stochastic methods for modeling and predicting complex dynamical systems. Cham: Springer; 2023.

Zeng Z, Lin C, Peng W, Xu M. The evolving paradigm of reliability engineering for complex systems: a review from an uncertainty control perspective. Aerospace. 2026; 13(2): 183. https://doi.org/10.3390/aerospace13020183.

Wang S, Teng Y, Perdikaris P. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM J Sci Comput. 2021; 43(5). https://doi.org/10.1137/20M1318043.

Nooraiepour M. Traditional and machine learning approaches to partial differential equations: a critical review of methods, trade-offs, and integration. Preprints. 2025. https://doi.org/10.20944/preprints202509.0472.v1.

Xenopoulos P, Rulff J, Nonato LG, Barr B, Silva C. Calibrate: interactive analysis of probabilistic model output. IEEE Trans Vis Comput Graph. 2022; 29(1): 853-863.

https://doi.org/10.1109/TVCG.2022.3209489.

Saravana M, Roopa M, Arunalatha J, Venugopal K. Transformers for multivariate time series forecasting: comprehensive analysis, challenges, research opportunities and future prospects. IEEE Access. 2026.

https://doi.org/10.1109/ACCESS.2026.3654408.

Mortezanejad SAF, Wang R, Mohammad-Djafari A. Physics-informed neural networks with unknown partial differential equations: an application in multivariate time series. Entropy. 2025; 27(7): 682.

https://doi.org/10.3390/e27070682.

Zhao C, Zhang F, Lou W, Wang X, Yang J. A comprehensive review of advances in physics-informed neural networks and their applications in complex fluid dynamics. Phys Fluids. 2024; 36(10).

https://doi.org/10.1063/5.0226562.

Kathari S, Tangirala AK. A novel framework for causality analysis of deterministic dynamical processes. Ind Eng Chem Res. 2022;61(50):18426-18444. https://doi.org/10.1021/acs.iecr.2c02072.

Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L. Physics-informed machine learning. Nat Rev Phys. 2021; 3(6): 422-440.

Yazdani D, et al. Robust optimization over time: a critical review. IEEE Trans Evol Comput. 2023;28(5):1265-1285.

https://doi.org/10.1109/TEVC.2023.3306017.

Kovachki N, et al. Neural operator: learning maps between function spaces with applications to PDEs. J Mach Learn Res. 2023;24(89):1-97. Available from: http://jmlr.org/papers/v24/21-1524.html.

Azizzadenesheli K, Kovachki N, Li Z, Liu-Schiaffini M, Kossaifi J, Anandkumar A. Neural operators for accelerating scientific simulations and design. Nat Rev Phys. 2024; 6(5): 320-328.

https://doi.org/10.1038/s42254-024-00712-5.

Wang R, Yu R. Physics-guided deep learning for dynamical systems: a survey. ACM Comput Surv. 2025; 58(5): 1-31.

https://doi.org/10.1145/3766887.

Adombi AVDP. Scientific machine learning in hydrology: a unified perspective. Earth Sci Inform. 2025; 18(4): 522.

https://doi.org/10.1007/s12145-025-02021-6.

Soibam J, Aslanidou I, Kyprianidis K, Fdhila RB. Inverse flow prediction using ensemble PINNs and uncertainty quantification. Int J Heat Mass Transf. 2024; 226: 125480. https://doi.org/10.1016/j.ijheatmasstransfer.2024.125480.

Mienye ID, Swart TG, Obaido G. Recurrent neural networks: a comprehensive review of architectures, variants, and applications. Information. 2024; 15(9): 517. https://doi.org/10.3390/info15090517.

Özalp E, Margazoglou G, Magri L. Reconstruction, forecasting, and stability of chaotic dynamics from partial data. Chaos. 2023; 33(9). https://doi.org/10.1063/5.0159479.

Younesi A, Ansari M, Fazli M, Ejlali A, Shafique M, Henkel J. A comprehensive survey of convolutions in deep learning: applications, challenges, and future trends. IEEE Access. 2024; 12: 41180-41218.

https://doi.org/10.1109/ACCESS.2024.3376441.

Kang H, Kang P. Transformer-based multivariate time series anomaly detection using inter-variable attention mechanism. Knowl Based Syst. 2024; 290: 111507. https://doi.org/10.1016/j.knosys.2024.111507.

da Silva DMGFP. Transformers in time series forecasting: A systematic literature review. 2024. Available from: http://hdl.handle.net/10362/175056.

Shi Y, Wei P, Feng K, Feng DC, Beer M. A survey on machine learning approaches for uncertainty quantification of engineering systems. Mach Learn Comput Sci Eng. 2025; 1(1): 11. https://doi.org/10.1007/s44379-024-00011-x.

Allen S, Ziegel J, Ginsbourger D. Assessing the calibration of multivariate probabilistic forecasts. QJR Meteorol Soc. 2024; 150(760): 1315-1335. https://doi.org/10.1002/qj.4647.

Folgoc LL, et al. Is MC dropout Bayesian? arXiv Preprint. 2021; arXiv:2110.04286. https://doi.org/10.48550/arXiv.2110.04286.

Søndergaard HAN, Shaker HR, Jørgensen BN. Multi-method fault detection considering uncertainty through MC dropout for enhanced voting. Energy Build. 2026; 117082. https://doi.org/10.1016/j.enbuild.2026.117082.

Immer A, Palumbo E, Marx A, Vogt J. Effective Bayesian heteroscedastic regression with deep neural networks. Adv Neural Inf Process Syst. 2023; 36: 53996-54019.

Seitzer M, Tavakoli A, Antic D, Martius G. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. arXiv Preprint. 2022;arXiv:2203.09168. https://doi.org/10.48550/arXiv.2203.09168.

He H, et al. Robust multivariate time series forecasting against intraseries and interseries transitional shift. IEEE Trans Neural Netw Learn Syst. 2025. https://doi.org/10.1109/TNNLS.2025.3593156.

Russell RL, Reale C. Multivariate uncertainty in deep learning. IEEE Trans Neural Netw Learn Syst. 2021; 33(12): 7937-7943. https://doi.org/10.1109/TNNLS.2021.3086757.

Xiong D, Xu F. A robust diagnostic approach in mean shifts for multivariate statistical process control. J Stat Comput Simul. 2025; 95(3): 507-524. https://doi.org/10.1080/00949655.2024.2431856.

Shi Y, Beer M. Physics-informed neural network classification framework for reliability analysis. Expert Syst Appl. 2024; 258: 125207. https://doi.org/10.1016/j.eswa.2024.125207.

Weinans E, Quax R, van Nes EH, van de Leemput IA. Evaluating the performance of multivariate indicators of resilience loss. Sci Rep. 2021; 11(1): 9148. https://doi.org/10.1038/s41598-021-87839-y.

Li H, Sun C. PGVAE-VBAKF: a robust strategy for complex system response prediction and noise variance estimation considering modeling errors and nonstationary noises. Mech Syst Signal Process. 2026; 243: 113699.

https://doi.org/10.1016/j.ymssp.2025.113699.

Astro__pat. Dynamical system multivariate time series [Internet]. Kaggle. Available from: https://www.kaggle.com/datasets/patrickfleith/dynamical-system-multivariate-time-series-forecast

Behnoudfar P, Chen N. RL-DAUNCE: reinforcement learning-driven data assimilation with uncertainty-aware constrained ensembles. arXiv Preprint. 2025;arXiv:2505.05452. https://doi.org/10.48550/arXiv.2505.05452.

Chen Z, Badrinarayanan V, Lee CY, Rabinovich A. GradNorm: gradient normalization for adaptive loss balancing in deep multitask networks. In: Proceedings of the International Conference on Machine Learning. PMLR; 2018. p. 794-803.

Yu R, Wang R. Learning dynamical systems from data: an introduction to physics-guided deep learning. Proc Natl Acad Sci USA. 2024; 121(27). https://doi.org/10.1073/pnas.2311808121.

Rane N, Choudhary SP, Rane J. Ensemble deep learning and machine learning: applications, opportunities, challenges, and future directions. Stud Med Health Sci. 2024; 1(2): 18-41. https://doi.org/10.48185/smhs.v1i2.1225.

Kirsch A. (Implicit) ensembles of ensembles: epistemic uncertainty collapse in large models. arXiv Preprint. 2024;arXiv:2409.02628. https://doi.org/10.48550/arXiv.2409.02628.

Liang X, Liu Z, Wang J, Jin X, Du Z. Uncertainty quantification-based robust deep learning for building energy systems considering distribution shift problem. Appl Energy. 2023; 337: 120889.

https://doi.org/10.1016/j.apenergy.2023.120889.

Castán-Lascorz M, Jiménez-Herrera P, Troncoso A, Asencio-Cortés G. A new hybrid method for predicting univariate and multivariate time series based on pattern forecasting. Inf Sci. 2022; 586: 611-627.

https://doi.org/10.1016/j.ins.2021.12.001.

Han L, Ye HJ, Zhan DC. The capacity and robustness trade-off: revisiting the channel independent strategy for multivariate time series forecasting. IEEE Trans Knowl Data Eng. 2024; 36(11): 7129-7142.

https://doi.org/10.1109/TKDE.2024.3400008.

Downloads

Published

2026-06-26

Issue

Section

Articles