Abstract
Partial differential equations (PDEs) are instrumental for modeling dynamicalsystems in science and engineering. The advent of neural networks has initiateda significant shift in tackling these complexities though challenges inaccuracy persist, especially for initial value problems. In this paper, weintroduce the $\textit{Time-Evolving Natural Gradient (TENG)}$, generalizingtime-dependent variational principles and optimization-based time integration,leveraging natural gradient optimization to obtain high accuracy inneural-network-based PDE solutions. Our comprehensive development includesalgorithms like TENG-Euler and its high-order variants, such as TENG-Heun,tailored for enhanced precision and efficiency. TENG's effectiveness is furthervalidated through its performance, surpassing current leading methods andachieving machine precision in step-by-step optimizations across a spectrum ofPDEs, including the heat equation, Allen-Cahn equation, and Burgers' equation.