Hyperparameter optimization in neural networks approach to singular matrix differential systems

Venkata Sundaranand Putcha; Praveen Kumar Ulebeedu Matam

doi:10.5269/bspm.78566

Venkata Sundaranand Putcha Rayalaseema University
Praveen Kumar Ulebeedu Matam Department of Mathematics, Rayalaseema University, Kurnool-518007, AP, INDIA

Résumé

Singular Matrix Differential Systems (SMDS) or, alternatively, semi-state, degenerate, descriptor, constrained, or differential-algebraic systems (DAEs) are key to modeling dynamic processes experiencing abrupt structural or parametric changes. It becomes difficult to solve initial value problems (IVPs) for these systems as classical methods are ineffective to apply owing to singularity and a lack of closed-form solutions. This paper introduces an adaptive neural network solution to linear singular matrix differential systems (LSMDS) with a semi-supervised learning framework. Singular systems, a commonplace within engineering models and constraint-laden physical models, are extremely computationally challenging with stiffness, index intricacy, and inconsistent initial conditions. Standard numerical solvers are afflicted by similar challenges, especially with singular matrices. To compensate, we propose a hybrid neural structure joining (i) a systematic search with activation functions and (ii) a two-stage optimizer sequence joining Adam's strengths with those of L-BFGS. The structure learns precise approximations without mesh-based discretizations. We build a task-specific loss function comprising differential-algebraic systems (DAEs) to guide optimizer training. We also undertake a detailed hyperparameter study, comprising network depth, width, activation function choosing, and optimizer switching plans, to establish suitable configurations. We evaluate our approach with a number of benchmark singular systems, achieving better accuracy, robustness, and generalization beyond standard solvers. This paper provides a flexible, data-efficient substitute to solving challenging constraint-laden systems, with significant applications to scientific computation and real-world modeling.

Téléchargements

Les données sur le téléchargement ne sont pas encore disponible.

Biographie de l'auteur

Praveen Kumar Ulebeedu Matam, Department of Mathematics, Rayalaseema University, Kurnool-518007, AP, INDIA

Praveen Kumar UM is a Post Graduate in Mathematics and presently working for his Ph.D in the Department of Mathematics, Rayalaseema University, Kurnool-518007, AP, INDIA. His areas of interest include Dynamical Systems, Artificial Neural Networks and Cryptology.

Références

P.V.S.Anand and K.N.Murty, Controllability and Observability of Liapunov type matrix difference system, In Proceedings of 50th Congress of ISTAM (An International Meet) IIT Kharagpur, 125–132, 2005.

P.V.S.Anand, Controllability and Observability of the Matrix Lyapunov Systems, In Proceedings of the international conference on Recent Advances in Mathematical Science and Applications (RAMSA) held at Vizag, 117–131, 2009.

K. E. Brenan, S. L. Campbell, and L. R. Petzold, Numerical Solution of Initial-Value Problems in Differential-Algebraic Equations, SIAM, 1996.

S. L. Campbell, C. D. Meyer, and N. J. Rose, “Applications of Drazin Inverse to Linear System of Differential Equations with Singular Coefficients,” SIAM J. Appl. Math., vol. 3, pp. 411–425, 1976.

S. L. Campbell, “Linear System of Differential Equations with Singular Coefficients,” SIAM J. Math. Anal., vol. 6, pp. 1057–1066, 1977.

S. L. Campbell, Singular Systems of Differential Equations, Pitman, Boston, 1980.

S. L. Campbell, “High-index DAEs in applications,” Appl. Numer. Math., vol. 5, 1989.

L.N.Charyulu, Rompicharla, Venkata Sundaranand Putcha, G.V.S.R.Deekshithulu, Controllability and observability of fuzzy matrix discrete dynamical systems, Journal of Nonlinear Sciences and Applications, 12, 816–828, 2019.

L.N.Charyulu, Rompicharla, Venkata Sundaranand Putcha, G.V.S.R.Deekshithulu, Existence of (ϕ⊗ψ) bounded Solutions For Linear First Order Kronecker Product Systems, In International Journal of Research Scientific Journal, 11, Issue 06(E), 39047–39053, 2020.

L.N.Charyulu, Rompicharla, Venkata Sundaranand Putcha, G.V.S.R.Deekshithulu, Kronecker product Three point boundary value problems Existence and Uniqueness, In International Research Journal of Engineering and Technology (IRJET), 8, Issue:02, 2021.

D. Cobb, “On the Solution of Linear Differential Equations with Singular Coefficients,” J. Differential Equations, vol. 46, no. 3, pp. 310–323, 1982.

P. Van Dooren, “The Generalized Eigenstructure Problem in Linear Systems Theory,” IEEE Trans. Automat. Control, vol. 26, pp. 111–129, 1981.

P. Van Dooren, “The Eigenstructure of an Arbitrary Polynomial Matrix: Computational Aspects,” Linear Algebra Appl., vol. 50, pp. 545–579, 1983.

C. W. Gear, “Numerical Initial Value Problems in Differential-Algebraic Equations,” IEEE Trans., vol. 36, 1988.

T. Geerts, “Invariant Subspaces and Invertibility Properties for Singular Systems: The General Case,” Linear Algebra Appl., vol. 183, pp. 61–68, 1993.

A. Griewank, “On solving singular systems numerically,” Math. Comp., vol. 42, pp. 123–132, 1984.

G. E. Karniadakis et al., “Physics-informed machine learning,” Nature Rev. Phys., vol. 3, pp. 422–440, 2021.

P. Kunkel and V. Mehrmann, Differential-Algebraic Equations: Analysis and Numerical Solution, EMS, 2006.

P. Lancaster, “Explicit Solutions of Linear Matrix Equations,” SIAM Review, vol. 12, no. 4, pp. 544–566, 1970.

V. Lakshmikantham and D. Trigiante, Theory of Difference Equations – Theory, Methods and Applications, Academic Press, 1988.

J. Lee and A. Ryou, “Hyperparameters in PINNs,” Mach. Learn. Sci. Technol., vol. 3, no. 1, 2022.

D. Li et al., “DAE-PINN: Neural approach for constrained dynamics,” IEEE Trans. Neural Netw., vol. 34, no. 2, 2023.

J. J. Loiseau, “Some Geometric Considerations about the Kronecker Normal Form,” Internat. J. Control, vol. 42, pp. 1411–1413, 1985.

Y. Lu et al., “Deep learning of dynamics with constraints,” Neural Comput., vol. 33, no. 5, 2021.

K.N.Murty, K.R.Prasad, and P.V.S.Anand, Two-point boundary value problems associated with Lyapunov type matrix difference system, Dynamic Systems and Applications, USA, 4(2), 205–213, 1995.

K.N.Murty, P.V.S.Anand, and V.Lakshmi Prasannam, First Order Difference System Existence and Uniqueness, Proceedings of the American Mathematical Society, 125(12), 3533–3539, 1997.

L. Petzold, “Differential/algebraic equations are not ODEs,” SIAM J. Sci. Stat. Comput., vol. 3, pp. 367–384, 1982.

Putcha, V. S. (2014). Discrete linear Sylvester repetitive process. Nonlinear Studies, 21(2), 205–218.

M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems,” J. Comput. Phys., vol. 378, pp. 686–707, 2019.

P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for Activation Functions,” arXiv preprint arXiv:1710.05941, 2017.

Venkata Sundaranand Putcha, D.P.R.V.Subba Rao, and Ramu Malladi, Existence of ψ-Bounded Solutions for First Order Matrix Difference System on Z, In International journal of Embedded Systems and Emerging Technologies, 5(1), 6–16, 2019.

G. C. Verghese, B. Levy, and T. Kailath, “A Generalized State Space for Singular Systems,” IEEE Trans. Automat. Control, vol. 26, pp. 811–831, 1981.

J. Chen, J. Tang, M. Yan, S. Lai, K. Liang, J. Lu, and W. Yang, “Physical information neural networks for solving high-index differential-algebraic equation systems based on Radau methods,” arXiv preprint arXiv:2310.12846, 2023. Available: https://arxiv.org/abs/2310.12846

E. L. Yip and R. F. Sincovec, “Solvability, Controllability, and Observability of Continuous Descriptor Systems,” IEEE Trans. Automat. Control, vol. 26, pp. 702–707, 1981.

X. Wu, D. Zhang, M. Zhang, C. Guo, S. Zhao, Y. Zhang, H. Wang, and B. Yang, “AutoPINN: When AutoML meets physics-informed neural networks,” arXiv preprint arXiv:2212.04058, 2022. Available: https://arxiv.org/abs/2212. 04058

A. D. Jagtap and G. E. Karniadakis, “Adaptive activation functions accelerate convergence in deep and physics-informed neural networks,” J. Comput. Phys., vol. 404, p. 109136, 2020.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” in Adv. Neural Inf. Process. Syst., vol. 31, 2018.

C. Rackauckas et al., “Universal differential equations for scientific machine learning,” arXiv preprint arXiv:2001.04385, 2020.

Y. Wang, N. Patel, Z. Lu, L. Sun, and R. Yu, “Auto-PINN: Understanding and optimizing physics-informed neural architecture,” arXiv preprint arXiv:2205.13748, 2022.