Hyperparameter Optimization in Neural Networks Approach to Singular Matrix Differential Systems

Singular Matrix Differential Systems - Neural Networks

Abstract

Singular Matrix Differential Systems (SMDS) or, alternatively, semi-state, degenerate, descriptor, constrained, or differential-algebraic systems (DAEs) are key to modeling dynamic processes experiencing abrupt structural or parametric changes. It becomes difficult to solve initial value problems (IVPs) for these systems as classical methods are ineffective to apply owing to singularity and a lack of closed-form solutions. This paper introduces an adaptive neural network solution to linear singular matrix differential systems (LSMDS) with a semi-supervised learning framework. Singular systems, a commonplace within engineering models and constraint-laden physical models, are extremely computationally challenging with stiffness, index intricacy, and inconsistent initial conditions. Standard numerical solvers are afflicted by similar challenges, especially with singular matrices. To compensate, we propose a hybrid neural structure joining (i) a systematic search with activation functions and (ii) a two-stage optimizer sequence joining Adam's strengths with those of L-BFGS. The structure learns precise approximations without mesh-based discretizations. We build a task-specific loss function comprising differential-algebraic systems (DAEs) to guide optimizer training. We also undertake a detailed hyperparameter study, comprising network depth, width, activation function choosing, and optimizer switching plans, to establish suitable configurations. We evaluate our approach with a number of benchmark singular systems, achieving better accuracy, robustness, and generalization beyond standard solvers. This paper provides a flexible, data-efficient substitute to solving challenging constraint-laden systems, with significant applications to scientific computation and real-world modeling.