Abstract
The reliable prediction of the temporal behavior of complex systems is key in numerous scientific fields. This strong interest is however hindered by modeling issues: Often, the governing equations describing the physics of the system under consideration are not accessible or, when known, their solution might require a computational time incompatible with the prediction time constraints. Not surprisingly, approximating complex systems in a generic functional format and informing it exâ€“nihilo from available observations has become common practice in the age of machine learning, as illustrated by the numerous successful examples based on deep neural networks. However, generalizability of the models, margins of guarantee and the impact of data are often overlooked or examined mainly by relying on prior knowledge of the physics. We tackle these issues from a different viewpoint, by adopting a curriculum learning strategy. In curriculum learning, the dataset is structured such that the training process starts from simple samples toward more complex ones in order to favor convergence and generalization. The concept has been developed and successfully applied in robotics and control of systems. Here, we apply this concept for the learning of complex dynamical systems in a systematic way. First, leveraging insights from the ergodic theory, we assess the amount of data sufficient for apriori guaranteeing a faithful model of the physical system and thoroughly investigate the impact of the training set and its structure on the quality of longterm predictions. Based on that, we consider entropy as a metric of complexity of the dataset; we show how an informed design of the training set based on the analysis of the entropy significantly improves the resulting models in terms of generalizability and provide insights on the amount and the choice of data required for an effective datadriven modeling.
1 Introduction
In the present era of mathematization of nature, the need for modeling is ubiquitous. Focusing on engineering as an example, models are invaluable for predicting quantities of interest, designing and optimizing physical systems, understanding their behavior in an environment, allowing to derive control strategies. Physical models have historically been developed from a combination of observational data and first principles, depending on the amount and type of prior knowledge. This approach is still followed today where the mathematical structure of a model is typically provided and observational data are used to inform it, i.e., learn the coefficients associated with the different terms of the retained structure. Countless variants exist but they generally follow this philosophy. Nonetheless, in many relevant cases, the physics of the phenomena is very complex and the interplay of observation with physical intuition is not sufficient to sort out a formalized model, let alone more rigorous general laws. In these cases, it is tempting to resort to a pure datadriven approach, in which some model is built up entirely from data. The datadriven approach received some attention after the work of Takens [9, 35, 38], that provided the theoretical backbone to predictive models based on timeseries analysis of lowdimensional systems [20]. These approaches found new motivation following the significant progresses in the field of machine learning (ML), fostered by the massive augmentation of computerbased activities [15]. Even if a clear framework is still missing, the use of such algorithms to build models from data has been introduced in the last decades [see, e.g., 36, 43] and, thanks to the recent blooming of datadriven techniques, has been considered in the modeling a variety of different physical phenomena [see, e.g., 5, 8, 31]. However, it is important to stress that the introduction of deep learning approaches has not necessarily changed the global picture of modeling. The potential versatility of highly expressive neural networks is often associated with a large amount of necessary observational data to suitably inform the associated coefficients of the model. Yet, not every situation enjoys a massive amount of observational data and can afford to inform a widely generic model. In practice, it is more likely that data are scarce due the cost and/or technical difficulty in obtaining them and complex highdimensional systems may need a quantity of data significantly exceeding the observing capability, [3]. In that sense, the lack of data for inferring physical models might have direct consequences on the relevance of these highly expressive models. A strategy to alleviate this limitation is to simplify the postulated structure of the model so that it is less data hungry, with simpler models here understood as involving fewer parameters to be informed from the available data, hence a weaker constraint on the amount of training samples. Another strategy is to leverage prior knowledge one has on the physical system under consideration to guide the structure and the learning process, hence saving on data. For instance, known symmetries can be enforced by design in the structure or in the representation format of the input data, effectively reducing the need for data. A similar idea applies to invariances one may know the system obeys (conserved quantities such as energy, translational or rotational invariances) or properties such as stability in a given sense. This physicsaware machine learning approach has now become popular and is the motivation of numerous recent efforts [see [21, 33, 42, 46], among many]. In that sense, the modeling of chaotic dynamical systems is a natural playfield [4, 28]. Remarkably, previous works seem to indicate an intriguing capability to give accurate forecasting of chaotic systems, even relatively highdimensional [27], apparently improving over deterministic embedding techniques [20]. These results therefore deserve to be thoroughly analyzed. In the present work, we aim to shed some light on the challenges in datadriven learning dynamical systems and to explore the idea of leveraging prior information, here in the form of known organization of the dynamics, to inform the process of collecting relevant observations. Specifically, we adopt a curriculum learning strategy for dynamical systems from first principles and assess it with respect to common sampling strategies. Curriculum learning relies on the idea that machines, like humans, learn best if first exposed to â€śeasyâ€ť situations prior to dealing with â€śmore complexâ€ť ones when well into the learning process. A possible rationale is that curriculum learning can be interpreted as an adaptive regularization method for optimization of a nonconvex objective function. By progressively introducing more complex samples, the associated cost landscape evolves from nearconvex to a more complex function exhibiting many more local extrema. Sorting training examples in this way may hence favor the optimization process toward a better extremum. This concept has been mostly developed in robotics and control of systems, [34] and was formalized more recently in the context of machine learning, [2]. While it still lacks a firm theoretical ground despite some efforts [44], curriculum learning has been widely illustrated and successfully employed in a variety of applications, ranging from robotics to video games, from computer vision to natural language processing and reinforcement learning, [25, 37]. What makes a situation, or training sample in the context of machine learning, â€śeasyâ€ť or â€ścomplexâ€ť is casespecific. For instance, in classification problems, the complexity of a particular example presented to the machine may be associated with the distance to the decision surface (e.g., SVM) or with the amount of noise in the data. Here, we illustrate the impact of curriculum learningbased sampling strategies, by considering the celebrated Lorenzâ€™63 system. While deterministic and low dimensional, the Lorenz system features many challenges encountered in more complex configurations, such as sensitivity to initial conditions and an attractor dimension \(d_{\varvec{A}}\) significantly lower than the ambient dimension \(d\), here \(d_{\varvec{A}}\simeq 2.06 < d= 3\). It is thus somehow a minimally complex unit and justifies its relevance for illustrating some challenges in learning dynamical systems. We associate the complexity of a training element with its information entropy, in the sense of Shannon, and recognize that the solution of dynamical systems in the vicinity of an unstable fixed point is well described by a linear model and exhibits a low entropy. This observation is key and allows a principled sampling strategy in collecting observations of increasing complexity from the system at hand as it evolves from an unstable fixed point. We explore and compare different strategies for sampling a dynamical system in the aim of learning its behavior and predicting its state over time. These strategies only differ by the region in the phase space where observational samples are taken from, every other aspects being identical (sampling frequency, number of samples, observed quantity, etc.). These different configurations are chosen to be typical of situations ranging from no prior information, besides an estimated upper bound of the dimension of the dynamical system up to approximate knowledge of some invariants of the system. We carry out a comprehensive statistical analysis in order to systematically test the impact that different sampling strategies have on the quality of the resulting models in terms of prediction abilities and generalizability. From an algorithmic point of view, recurrent structures such as autoregressive models or recurrent neural networks are popular choices in such situations. In the present study, we focus on the LSTM (Long ShortTerm Memory) architecture [17] for its widespread use, including applications in prediction of dynamical systems behavior [11]. Moreover, it has been recently shown that the performance given by different architectures including reservoir computing is similar [41], although for lowdimensional systems like Lorenzâ€™63, LSTMs appear to give the best results. Given this choice, we also provide bestpractice guidance concerning the possible impact of initial conditions and the memory effects on the results of supervised learning models, beside analyzing the possibility of using less data while preserving generalizability and quality based on curriculum learning strategies. The paper is organized as follows. General principles in learning a system from data are briefly discussed in Sect.Â 2 to set up the stage and define notations. The general class of models we consider in this work is presented in Sect.Â 3. The specific configuration we retain for the present discussion is the Lorenzâ€™63 introduced in Sect.Â 4, together with the different sampling strategies. The results are gathered and presented in Sect.Â 5 before the discussion in Sect.Â 6 closes the paper.
2 Theoretical background
2.1 Learning dynamical systems
The present work lies in the framework of learning a model to approximate the time evolution of a state of the system at hand from observations \(\varvec{s}(t) \in \varvec{R}^{n_{s}}\). To do so, one needs a representation for the state and a dynamical model for describing its evolution in time. A representation of the state is given by a function \(g\) mapping observations \(\varvec{s}_\varGamma (t)\) over a measurement time horizon \(t \in \varGamma \), to a set of coordinates \(\varvec{v}\in \varvec{R}^{d_v}\). Several mappings have been proposed in the literature, including delays by time intervals \(\varDelta t\), \(\varvec{v}_i=\varvec{s}(ti\varDelta t)\) for \(i=0,1,\ldots \), and time derivatives of varying orders i, \(\varvec{v}_i=\varvec{s}^{(i)}\), [7, 13]. To obtain the size of the coordinate space, we assume that the state of the system \(\varvec{u}\) lies on an attractor \(\varvec{A}\) of dimension \(d_{\varvec{A}}< d\) that is a boundaryless bounded, smooth submanifold of \(\varvec{R}^{d}\). In this case, the Whitney embedding theorem [45] shows that if the number of coordinates \(d_v\) is such that \(d_v> 2 \, d_{\varvec{A}}\), then the dynamics of the state can be entirely captured by the new coordinate system. The Takens embedding theorem [35, 38] provides a way to make this result practical by asserting that one can take this number of time delays. Recent results have extended this theorem by providing guaranteed recovery properties based on the measurement operator, type of attractor, and other parameters of the problem, [10]. In this work, we assume no prior knowledge of the governing equations of the model and rely on a purely datadriven approach. The dynamical model for describing the time evolution of \(\varvec{v}(t) \in \varvec{R}^{d_v}\) can be formulated in various ways, including feedforward neural networks, owing to their high expressiveness. As an alternative, for instance in case an upper bound of the attractor dimension is unknown due to process noise or measurement noise, a recurrent model \(F_{\varvec{\theta }}\) can be employed for describing the dynamics of \(\varvec{v}\). The model is parameterized by \(\varvec{\theta }\in \varvec{R}^{n_{\varvec{\theta }}}\) and writes
where a dotted quantity denotes its timederivative. Learning a model for the system under consideration takes the form of a classical supervised learning problem based on the misfit between observations \(\varvec{s}_\varGamma \equiv \varvec{s}\left( \varvec{u}\left( t \in \varGamma \right) \right) \) collected through a measurement operator \(h: \varvec{u}\mapsto \varvec{s}\) over a time domain \(\varGamma = \left[ t_0, t_T\right] \) and predictions, in the sense of some norm \(\left\ {\cdot }\right\ _{\varGamma }\). It typically results in an optimization problem in terms of the model parameters, possibly including a regularization term \(r\):
2.2 Collecting data
The main issue impacting the predictions based on observations is the quantity of data actually needed to approximate the dynamics in a given sense. The issue is related to the ergodic theory of dynamical systems, whose founding idea is that the longtime statistical properties of a deterministic system can be equivalently described in terms of the invariant (timeindependent) probability, \(\mu \), such that \(\mu (S)\) is the probability of finding the system in any specified region S of its phase space. If the trajectories of a \(d\)dimensional ergodic system evolve in a bounded phase space \(\varOmega \subset \varvec{R}^{d}\), the PoincarĂ© recurrence theorem [30] ensures that analogs exist as it proves that the trajectories exiting from a generic set \(S\in \varOmega \) will return to such set S infinitely many times. The theorem holds for any points in S, almost surely. It was originally formulated for Hamiltonian systems, but it can be straightforwardly extended to dissipative ergodic systems provided initial conditions are chosen on the attractor. In \(d\)dimensional dissipative systems, the attractor \(\varvec{A}\) typically has a dimension \(d_{\varvec{A}}< d\). The probability \(\mu \left( B^{d}_{\textbf{x}}(\varepsilon ) \right) \) of finding points on the attractor which are in the \(d\)dimensional ball of radius \(\varepsilon \) ( with resolution \(\varepsilon \ll 1\)) around \(\textbf{x}\in \varvec{A}\subset \varvec{R}^d\):
When \(d_{\varvec{A}}\) is not an integer, the attractor and its probability measure are fractal. The attractor is typically multifractal [26], but we can assume without lack of generality that it is homogeneous. Given that the PoincarĂ© theorem proves the existence of good analogs, the issue is shifted to finding out how much time is needed for a PoincarĂ© cycle to end. This key problem was considered by Smolouchoski and resolved by Kac [19], who showed that for an ergodic system, given a set \(S \equiv B^{d}_{\textbf{x}^\star }(\varepsilon )\), the mean recurrence time \(\langle \tau _S(\textbf{x}^\star ) \rangle \) relative to S is inversely proportional to the measure of the set S, i.e.,
where the average \(\langle \cdot \rangle \) is computed over all the points \(\textbf{x}\in S\) according to the invariant measure. Therefore, if we require \(\varepsilon \) to be small and if \(d_{\varvec{A}}\) is large, the average recurrence time becomes huge, a symptom of the curse of dimensionality in datadriven methods.
3 Machine learning approach
3.1 Architecture
As discussed above, a set of \(d\)dimensional observations retained to express a state of the system under consideration might not be a state vector, in the sense that it does not necessarily contain enough information to uniquely define its future evolution as an autonomous deterministic dynamical system. Even though the system were Markovian, it might not be so in the retained representation and may implicitly involve latent variables. This is typically the case in situations of partial observability or when measurements are affected by noise. As discussed in Sect.Â 2, under conditions rigorously studied in the embedology line of research, [9, 35], latent state variables can be substituted with observations over a recent past, that is, temporal information is traded for unavailable highdimensional phase space information. In the present work, we consider the LSTM (Long ShortTerm Memory) architecture. Compared with standard multilayer perceptrons, LSTMs additionally feature internal variables \(\varvec{\zeta }\) carriedover through the recurrence and effectively acting as a â€śmemory.â€ť Training an LSTM then allows to derive a discreteintime model predicting the future â€śstateâ€ť \(\varvec{v}_{k+1} \equiv \varvec{v}\left( t_k + \varDelta t\right) \) from the current state and memory:
The retained architecture comprises 2 LSTM layers of 50 neurons each and one fully connected layer, with tanh activation functions in the LSTM cells and sigmoids for the state output (see the sketch in Fig.Â 1). The present architecture was deemed as a good compromise between accuracy and computational expenses, after an analysis with comparable architectures with one and three layers has been performed. A 1step ahead model is learned based on a whole trajectory, or a set of trajectories, in the \(L^2\)sense. The training is performed using the ADAM optimizer and an adaptive learning rate from 0.01 to \(10^{5}\), with a decay factor of 0.07. The maximum number of epochs is limited to 1000.
3.2 Strategies for sampling
Learning a model here consists in estimating the best set of parameters \(\varvec{\theta }\) in the sense of the optimization problem defined in Eq.Â 2. In the context of neural networks, this acts as the loss function used for training. The regularization term may promote several properties of the optimal solution \(\varvec{\theta }\), such as the popular choice of low magnitude (ridge regression), \(r\left( \varvec{\theta }\right) \sim \left\ {\varvec{\theta }}\right\ _{2}\), or sparsity, \(r\left( \varvec{\theta }\right) \sim \left\ {\varvec{\theta }}\right\ _{1}\), possibly mimicked with a suitable dropout strategy. We adopt a discreteintime viewpoint, in a formulation consistent with the discrete model in Eq.Â 5. The loss function \(\mathscr {L}\) associated with the training problem then writes
with K such that \(T = K \, \varDelta t\) and the 1step ahead dynamical model \(\displaystyle \varvec{v}_{k+1} = F_{\varvec{\theta }}\left( \varvec{v}_k; \varvec{\zeta }_k\right) \). The loss is here simply defined in terms of the Euclidean distance but many alternative definitions could be employed. For instance, optimal transportbased metrics, such as Dynamic Time Warping (DTW), can prove useful when observations come at a poorly known or controlled time pace. While the way the prediction from the model is compared with observational data is pivotal, the focus in this work is instead on the relevance of the training data. The present findings are believed to hold, disregarding the definition of the loss. A standard \(L^2\) distance, implicitly assuming regularly sampled measurements, is hence retained. The measurement operator is the identity, \(h\equiv I\), so that the measurements are associated with the state of the system, \(\varvec{s}_k \equiv \varvec{u}_k\), \(\forall \, k\). No noise affects the observations, so that one is in a full observability situation. In this context, the observed system is Markovian in the representation space of the observations. However, a recurrent architecture is here considered as a general structure for learning, relevant for a wide class of situations. It further has the benefit of improving upon the robustness of the learning by virtue of some degree of redundant representation. Without loss of generality, we disregard the regularization term \(r\left( \varvec{\theta }\right) \) in the definition of the loss function which finally writes
While this context is favorable for learning (lowdimensional deterministic system, noiseless observations, full observability, etc.), it will be seen below that the measurement strategy still significantly affects the learning performance.
3.3 Curriculum learning, metrics and complexity of the datasets
As mentioned in the Introduction, we aim at verifying the extent to which a measurement strategy inspired by the curriculum learning principle can be applied to datadriven learning of dynamical systems and how this impacts the generalizability and quality of the resulting models. The rationale is the adaptive regularization of the objective function when moving from â€śsimplerâ€ť cost landscapes associated with simpler data toward more complex ones. From this perspective, any indicator of complexity can be used; here, we consider entropy as a measure and compute it based on singular value decomposition (SVD) [40]. Given an embedding for the measurements \(\varvec{s}\) based on time delaying
with k an integer and \(\tau \) the time delays, the matrix \(\varvec{S}\) is defined as
with \(n_e\) indicating the number of \(\tau \). The SVD is performed on the matrix and M singular values \(\left\{ \sigma _i^\textrm{SVD}\right\} _i\) are obtained such that the entropy H is defined as
with \({\overline{\sigma }}^\textrm{SVD}_i = \sigma ^\textrm{SVD}_i / \sum _{j=1}^M \sigma ^\textrm{SVD}_j\). The SVD entropy provides an estimation of the number of eigenvectors that are needed for an adequate representation of the dataset. In that sense, it measures the dimensionality of the data. With this metric, we will assess in the next section the complexity characterizing each of the training datasets that will be compared.
The second metric introduced in the article for the evaluation of the models is the correlation dimension, computed using the algorithm by Grassberger & Procaccia [16]. Considering a ball of radius \(\varepsilon \), the probability of finding a point scales as \(C(\varepsilon ) \sim \varepsilon ^{d_2}\), where \(d_2\) is an indication of the dimensionality. Using the Euclidian norm, the probability \(C\left( \varepsilon \right) \) can be efficiently computed as the correlation sum
with \(\varPhi \) the Heaviside step function, \(\varvec{s}\) the components of the embedded variable ranging \(i,j=1,2\dots N\), with a fixed time increment; N is the number of pairs. Due to the divergence of trajectories, most pairs \((s_i,s_j)\) with \(i\ne j\) are usually uncorrelated. Here, the correlation dimension \({d_2}\) is considered for the information provided on the fractal dimension of the attractor as well as the connections with the intrinsic dimension, a metric often found in machine learning [6]. Alternatively, the Lyapunov exponents can also be adopted as criteria.
4 Learning the Lorenzâ€™63 system
4.1 Lorenzâ€™63 system
To illustrate the different learning strategies, we consider the celebrated Lorenzâ€™63 system [24], originally introduced to mimic the thermal convection in the atmosphere. It consists of three coupled nonlinear ordinary differential equations (ODEs):
with \(\left\{ \sigma , \rho , \beta \right\} \in \varvec{R}^+.\) The system of ODEs is integrated in time with a \(4^{th}\) order Rungeâ€“Kutta scheme and a time step \(\varDelta t = 0.01\). Constants \(\sigma , \rho , \beta \) are positive parameters, related to dimensionless scales such as the Prandtl and the Rayleigh numbers. For \(\rho > \sigma \left( \sigma +\beta +3\right) /\left( \sigma \beta 1\right) \) and \(\sigma >\left( \beta +1\right) \), the solution \(\varvec{u}:= \left( x,y,z\right) \) exhibits a chaotic behavior. It is important for the following to note that the system features three unstable fixed points \(\varvec{u}_{\textrm{FP}}\), associated with a vanishing timederivative:
In the following, we shall consider the parameters fixed to \(\beta =8/3, \sigma =10, \rho =28\) as in the original work, such that the solution is in the chaotic regime. In this regime, the KaplanYorke dimension of the attractor is \(d_{\varvec{A}}\simeq 2.06\).
4.2 Setup of the learning method
We turn our attention to the training set \(\mathscr {T}= \left\{ \varvec{s}_k\right\} _{k=1}^K\) for learning the Lorenzâ€™63 system. It is a crucial ingredient of the learning process and largely conditions the quality of the resulting model. We consider four different training sets (\(\mathscr {T}_\textrm{ergo}\), \(\mathscr {T}_\textrm{split}\), \(\mathscr {T}_\textrm{ens}\), and \(\mathscr {T}_\textrm{FP}\) ) of identical settings (sampling frequency, number of samples, measurement operator, etc.) but following different sampling strategies.
4.2.1 Ergodicitycompliant training sets, \(\mathscr {T}_\textrm{ergo}\) and \(\mathscr {T}_\textrm{split}\)
The \(\mathscr {T}_\textrm{ergo}\) dataset is a long trajectory on the attractor. We consider it the reference case as it provides sufficient statistics for learning a reliable and accurate model. With \(\varepsilon = 10^{2}\) and the KaplanYorke dimension \(d_{\varvec{A}}\simeq 2.06\) of the Lorenzâ€™63 for the retained settings \(\left\{ \sigma ,\rho ,\beta \right\} \), [22], the required number of samples resulting from a constant sampling period of \(\varDelta t = 0.01\) is about 27,Â 000 strobes in time. A second dataset, \(\mathscr {T}_\textrm{split}\), is derived from \(\mathscr {T}_\textrm{ergo}\): It consists of one, long trajectory on the attractor, yet split in nine chunks of equal size and reshuffled, such that each chunk is thus composed by 3000 samples. As it will be clear when discussing the results (Sect.Â 5), this dataset will allow us to investigate some effects of memory.
4.2.2 Structured datasets: leveraging the fixed points, \(\mathscr {T}_\textrm{FP}\)
The training set \(\mathscr {T}_\textrm{FP}\) exploits prior knowledge on the structure of the system at hand, namely the fixed points. As mentioned in the Introduction, it is well known that dynamical systems are organized around critical features, such as fixed points, and that invariant sets provide information on the behavior of the solution, [14]. Besides fixed points, these features include higherdimensional invariant sets such as slow invariant manifolds. The sampling strategy here consists in acquiring observations from the system originating from its (unstable) fixed points, in contrast with other training sets obtained from the attractor. Specifically, three 3000sample trajectories are acquired from each of the three fixed points of the Lorenzâ€™63 system. These trajectories originate from a small deviation from each fixed points along each of the \(d=3\) directions given by the eigenvectors of the local Jacobian matrix. Note that, if the Jacobian is not available, a set of orthonormalized deviations can also be used. The resulting training set then consists of \(3 \times 3 \times 3000 = 27,000\) samples, consistent in size with the other training sets illustrated in Fig.Â 2.
4.2.3 Random sampling, \(\mathscr {T}_\textrm{ens}\)
Finally, we consider the sampling of short trajectories on the attractor. Nine trajectories are considered, each originating from a state randomly chosen on the attractor. Each trajectory is 3000 sample long as for the other strategies introduced above so that the total size of the training set is again \(9 \times 3000 = 27,000\) samples. This training set is representative of the widespread situations where information about a system come in several independent glimpses, as resulting from several measurement campaigns. It is illustrated in Fig.Â 2.
4.2.4 Entropy characterization
Using the definition of SVD entropy provided Eq.Â (10) with order \(n_e=3\) and timedelay \(\tau =1\), we characterize in the following each of the training sets. The SVD entropy algorithm has been chosen after a comparison among different strategies for the entropy estimation. First, we considered convergence of the estimates over long trajectories emanating from different initial conditions, in order to evaluate the consistency of the results in ergodic datasets; second, a validation based on entropy estimators requiring a reduced number of data, namely the Approximate Entropy algorithm by [29], has been performed. Good agreement was found between the different entropy estimators. From the physical viewpoint, the entropy for a time series is defined on the probability transition between the embedded states (unfolded through SVD). In Fig.Â 3, the entropy distributions are compared for each of the training sets: Quite neatly, the dataset \(\mathscr {T}_\textrm{FP}\) is associated with the lowest entropy, while the remaining datasets are all characterized by the same levels. From the physical viewpoint, this behavior is explainable by considering the higher predictability of the dynamics in the vicinity of the fixed points, where the behavior of the states close to them is well approximated by the linear limit. High predictability corresponds to less required information. As we will show in the following, this leads to an apparent contradiction: Despite less information is contained in the structured datasetsâ€”in a Shannon senseâ€”the possibility of training LSTM with well structured datasets characterized by increasing entropy in time allows to deploy a curriculum learning strategy.
5 Results
We now assess the quality of the models learned from various observation datasets of the Lorenzâ€™63 system.
5.1 How much training information?
The influence of the amount of information to train a model is first considered. Two models with the same architecture (cf. SectionÂ 3) are trained from two different datasets. These datasets consist of observations of the solution of the Lorenz system along the same trajectory on the attractor, but of different length. One dataset, \(\mathscr {T}_\textrm{ergo}\), comprises 27,000 samples, while the second one \(\mathscr {T}_\textrm{short}\) is its restriction to the first 3000 samples, so that \(\mathscr {T}_\textrm{short}\subset \mathscr {T}_\textrm{ergo}\). Once trained, the resulting models, hereafter, respectively, termed \(F_\textrm{ergo}\) and \(F_\textrm{short}\), are evaluated in terms of their longterm prediction capability. Results are gathered on Fig.Â 4. The performance of the models is here evaluated when predicting from a known (i.e., contained in their training set) and unknown (not contained) initial condition, to assess the generalization capability of the models. For all cases, memory is initialized to zero.
A known initial condition is first considered. As can be seen from the top row (left column) where it is plotted against the truth (red), the \(F_\textrm{short}\) model allows a reasonable quality of the prediction (plotted in blue) over a significant time horizon of about 6 Lyapunov times (7 time units). Since training the models is achieved via a stochastic procedure, e.g., stochastic gradient descent and random initialization of the parameters, and the landscape of the loss function is not globally convex, the underlying optimization problem ends up in a local optimum and bears an aleatoric contribution. To strengthen the above conclusions to account for the statistical uncertainty of neural network modeling, we hence consider an ensemble of 100 identical independent models and evaluate their performance [1]; the final evaluation is thus based on an aggregate metric that allows to provide a robust assessment of the relative performance across different trainings based on the different datasets. The right column of Fig.Â 4 shows the mean and standard deviation of the prediction of the ensemble of models for the same conditions as in the left column. From the top row, one may deduce that the prediction of the ensemble of models is very consistent and that the learning is robust. With the same short training set \(\mathscr {T}_\textrm{short}\), the trained model is now evaluated on an initial condition not contained in the training data, see Fig.Â 4 middle row. Compared with the previous situation (top row), the performance is seen to deteriorate, both for a single model (left column) and in average (right). In particular, the time horizon before the prediction significantly differs from the truth is much shorter, indicating a poor generalizability properties. When more training data are used, \(\mathscr {T}_\textrm{ergo}\), the resulting model remains accurate over a long time horizon even when predicting from an unknown initial condition, as seen from Fig.Â 4 (bottom row). Again, this is a reliable finding as evidenced from the ensemble performance seen on the right column.
In Fig.Â 5, a complementary analysis is shown, considering the average cumulative 2norm error over a short time period of 1 time unit. The averaging is performed considering 100 models for each of the four sampling strategies. The models are trained using random initialization of the memory. In the inset (a), the prediction emanating from trajectories included in the training set is shown. It is possible to see that the error is bounded and constant, at least within the short time window we consider here; the ergodic case is characterized by higher error due to a larger number of strobes backpropagated during the training in each batch. In (b), the error is shown for the testing sets. Firstâ€”coherently to Fig.Â 4â€”the error increases, while time increases; secondly, larger deviations are observed for the random dataset, while the cumulative error of the other cases is more comparable.
These results illustrate the fact that the training data, relied on to learn a model, have to be informative enough of the system under consideration to allow for generalizable models to be inferred. In particular, the superior performances shown in the top row of Fig.Â 4 and the low error shown in Fig.Â 5a, can be attributed to overfitting, as the initial conditions are contained in the learning dataset.
5.2 Relevance of the data
Besides the mere size of the training set, a critical aspect is its relevance for informing a model in view of a given objective function. To shed light on this pivotal aspect, the four different training sets introduced in Sect.Â 3.2, namely the ergodic (\(\mathscr {T}_\textrm{ergo}\)), the ergodicsplit (\(\mathscr {T}_\textrm{split}\)), the random sampled (\(\mathscr {T}_\textrm{ens}\)) and the structured based on the fixed points (\(\mathscr {T}_\textrm{FP}\)), all of the same size of 27,Â 000 samples, and analyzed using the SVD entropy in Sect.Â 4.2.4, are now evaluated in terms of the quality of the associated resulting model. To avoid any bias due to the initialization of the parameters, an ensemble of 100 models is trained for each situation and the comparison is made in terms of the ensemble performance [1]. A first metric to quantify the quality of the resulting model is the \(d_2\)dimension of the predicted model, computed by estimating the correlation sum in Eq.Â 11. This criterion is not associated with a given initial condition and is affected by the way the trained model consistently reproduces the fractal nature of the Lorenzâ€™63 system. Results are gathered in Fig.Â 6 in terms of a histogram of the estimated \(d_2\) dimension for the ensemble of 100 models for each of the four training sets. For a more quantitative appreciation, a threshold is defined when the dimension resulting from the trained model differs by more than 25 % from the truth \(d_{\varvec{A}}= 2.06\). As expected, the ergodicitycompliant training set \(\mathscr {T}_\textrm{ergo}\) leads to a good performance, with most models associated with a \(d_2\) dimension close to the true value and a few degenerate models such that \(d_2 \approx 0\). Only 14 % of the models are beyond the threshold. Models trained from \(\mathscr {T}_\textrm{FP}\) are also seen to achieve a similar level of performance. Again, almost all these models are associated with a rather good \(d_2\) dimension, with very few degenerate models. Only 8% of the models are beyond the threshold, an even better performance than models trained from \(\mathscr {T}_\textrm{ergo}\). In contrast, the \(\mathscr {T}_\textrm{ens}\) (random) and \(\mathscr {T}_\textrm{split}\) (split) dataset leads to a rather poor performance model, with a significant part of the models associated with \(d_2 \approx 0\) and, respectively, 34 and 26 % of the models beyond the threshold. An observation is in order in this regard. The models with \(d_2 \approx 0\) are all characterized by being stationary: The dynamics of the system stabilizes toward the fixed points after a short transient. On the other hand, only a limited number of models is characterized by \(d_2\approx 1\); in this case, the temporal dynamics settles on periodic orbits. Interestingly, in most of the stable models, the fixed points are correctly predicted, also for the random datasets where they are not included explicitly; however, their stability features are not correctly predicted, as already stressed.
It may be not quite surprising that models trained with \(\mathscr {T}_\textrm{ens}\) and \(\mathscr {T}_\textrm{split}\) are associated with a similar performance. Since the only difference between the two datasets is that the samples in \(\mathscr {T}_\textrm{ens}\) originate from a collection of 9 independent segments of trajectory, randomly located on the attractor, while \(\mathscr {T}_\textrm{split}\) is made of 9 contiguous segments. Yet, such behavior for the \(\mathscr {T}_\textrm{split}\) dataset appears to be not trivial, because it contains exactly the same information of the \(\mathscr {T}_\textrm{ergo}\) one about the phase space. Further assessment of these observations can be made using a different technical tool, namely tdistributed stochastic neighbor embedding (tSNE) of the parameters, shown in Fig.Â 7a. It is essentially a dimensionality reduction technique convenient for assessing highdimensional quantities [39]. The model parameters vectors are here mapped onto a twodimensional space where each model can hence be represented in terms of its corresponding coefficients \(\gamma _1\) and \(\gamma _2\). \(F_\textrm{ergo}\) and \(F_\textrm{FP}\) are mostly clustered in the center of the points cloud, indicating that, while the models are different, they are however similar to each other in terms of parameter distribution. Instead, the models obtained from the random sampling strategy (\(\mathscr {T}_\textrm{ens}\)) and the ones obtained stacking chunks from an ergodicitycompliant trajectory (\(\mathscr {T}_\textrm{split}\)) are distributed in the outer region. LSTM parameters are initialized as independent realizations of a Gaussian random variable, whose tSNE representation is typically visually close to a Gaussian as well since it penalizes deviation from a tStudent distribution. Recent works have shown that neural networks, associated with parameters weakly evolving during training while yet able to significantly improve the loss function, are often associated with good generalization properties, [18]. In the present study, the tSNE representation of models trained from the ergodicitycompliant and the structured datasets both appear close to a Gaussian distribution, a further indication that they might enjoy good generalization properties. In contrast, the tSNE distribution of the models trained from \(\mathscr {T}_\textrm{ens}\) or \(\mathscr {T}_\textrm{split}\) appears to deviate significantly from their initial distribution, which corroborates well with their poor performance. This distribution indicates that the best models are robust, in the sense of having similar parameters. Furthermore, training using information about fixed points appears equivalent to that based on a long trajectory. This observation is also quantified in terms of a radial distribution, see Fig.Â 7b. Both the models trained from \(\mathscr {T}_\textrm{ergo}\) and \(\mathscr {T}_\textrm{FP}\) present a distribution with highest density around zero, while \(\mathscr {T}_\textrm{ens}\) and \(\mathscr {T}_\textrm{split}\) lead to a bias in the parameter distribution, hence being associated with poor models. In view of the entropy levels associated with each of the datasets (see Sect.Â 4.2.4), we conclude that organizing the datasets around the fixed points, and leveraging the simpler, linear dynamics characterizing the vicinity of those points corresponds to deploying a curriculum learning for the training of fully datadriven models. We postulate that a possible rationale behind the superior performance can indeed be understood by a simpler landscape of the cost associated with the data obtained where the dynamics is well approximated by linear behavior. On the other hand, the curriculum learning assumption does not fully explain the discrepancies arising between models thatâ€”in principleâ€”are qualitatively the same (namely, \(\mathscr {T}_\textrm{ergo}\) and \(\mathscr {T}_\textrm{split}\)). For this reason, we turn our attention to a crucial element: the role of the memory in the recurrent models.
5.3 Handling the memory
Memory is considered in this section. Instead of the null memory (initialized to zero) considered so far and classically found in many applications, an initial memory chosen at random, drawn from a multivariate Gaussian distribution \({\mathcal {N}}(\varvec{0}, {\textbf{I}})\), is used. Compared to the null initial memory, the performances are seen to significantly increase for all models, as evidenced in Fig.Â 8. This is consistent with the fact that a nonzero hidden state improves the numerical conditioning of the training steps and prevents degenerate gradients, [12]. The contrast in performance between models learned from \(\mathscr {T}_\textrm{ergo}\) and \(\mathscr {T}_\textrm{split}\), which rely on the same information, leads to identify a key point. The LSTM model trained with \(\mathscr {T}_\textrm{ergo}\) learns to best fit one long trajectory. In the case of \(\mathscr {T}_\textrm{split}\), it learns to best fit nine short trajectories, which however, when stitched together, match the \(\mathscr {T}_\textrm{ergo}\) long trajectory. Yet, their performances are consistently very different in the null initialization situation, emphasizing that the initial memory of the retained recurrent network (LSTM) strongly affects the resulting performance. In the case of \(\mathscr {T}_\textrm{ergo}\), the impact of an incorrect initial memory is diluted within a long trajectory and weakly affects the learning quality. When short trajectories are considered instead, the relative impact of an incorrect initialization is stronger and can introduce a significant bias, hence resulting in a poor model. This observation is further supported by the model learned from \(\mathscr {T}_\textrm{FP}\). While it also relies on samples originating from nine distinct, non sequential, segments, its performance is good, as evidenced by Fig.Â 6. What is unique about this particular training set is that the samples are from segments of trajectories originating from (unstable) fixed points, thus associated with a locally linear dynamics. This feature allows the incorrect initial memory to be progressively wiped out in time without significantly affecting the predicted dynamics.
5.4 Can we use less data?
The results shown so far lead to slightly counterintuitive conclusions: On one hand, low entropy indicates that the dynamics described by the datasets \(\mathscr {T}_\textrm{FP}\) lead to a simpler training; on the other hand, simpler dynamics means high predictability and thus less information. In that sense, one might wonder whether it is possible to leverage these considerations for reducing the amount of data used during the training: In fact, the linear regions are less informative as they are redundant, in particular, when we consider â€śslowâ€ť trajectories emanating from the fixed points located in the two lobes of the Lorenzâ€™63 attractor. In order to provide more effective datasets, while preserving the curriculum learning idea, we consider here shortened trajectories where the initial samples are removed. The resulting characterization of the shortened datasets, based on the SVD entropy, is contained in Fig.Â 9: The entropy increases progressively as the dataset contains (relatively) more data stack in the nonlinear region. By considering these shortened datasets, we train LSTM models and assess their robustness using the \(d_2\) metric applied in the previous sections and study the associated models. As shown in Fig.Â 10, it is found that we can reduce the number of samples to a total of \(9 \times 2000 = 18,000\), as the behavior is essentially the same as for the original \(\mathscr {T}_\textrm{FP}\) models, i.e., not prone to overfitting and with low probability to produce biased results. When further reducing the data, results start to worsen until they definitively deteriorate for the last case \(9 \times 1000 = 9000\). Nonetheless, we can conclude that a reduction of \(33\%\) of the samples does not lead to loss of performance for the analyzed case. While the focus of the analysis is on data selection and their impact on the quality of performance, it is worth noting that any constraint applied to the cost function can further reduce the amount of data while preserving the performance of the system, for instance by explicitly introducing information about the stability of the system (Lyapunov exponents or the Jacobian matrix) or introducing multiobjective cost functions [23, 32]. In that sense, a combination of physics constraints, data selection/distillation and architecture optimization can deeply improve the prediction abilities of datadriven models with a significant reduction in the data necessary for training.
6 Concluding discussion
In this work, we apply the concept of curriculum learning for datadriven modeling of a complex dynamical system: This concept is inspired from human behavior and is based on the idea that better learning is achieved when machines are trained using data of increasing complexity. Among the possible explanations for motivating the effectiveness of this learning strategy, a possible one is related to the shape of the cost landscape, that evolves from being nearconvex to more complex as the complexity of the data increases. By sorting the data in this way, the optimization process is favored toward better extrema. Motivated by this methodology, we have performed a thorough analysis of some aspects inherent to the process of learning a complex physical system from observational data using the classical Lorenzâ€™63, owing to its simplicity and hence the possibility to carry out an exhaustive campaign of simulations. As a learning method, we have considered LSTM neural networks, an established recurrent structure often used in such a context. We addressed the following issues:

1.
Is there a minimum amount of data needed to obtain a robust model able to generalize?

2.
Is it possible to go beyond this limitation thanks to some â€śdatabasedâ€ť knowledge of the system using a metric?

3.
What is the impact of the initial state of the retained model on its resulting performance?
The first two questions are rather fundamental. In particular, the first question concerns the amount of data required for training and it is related to what is known for classical embedding approaches, which is bound to fail when timeseries data are not enough to reconstruct the entire phase space. Generally speaking, in absence of prior information on the system at hand besides an estimated upper bound of its dimension, the ergodic theory, and notably the Kac lemma, provides a reliable criterion on the minimal amount of data necessary to train a model. This amount of data grows exponentially with the dimension of the attractor. Models trained on a smaller amount of training data have been shown to consistently lead to poor performance, as illustrated both via the timeseries prediction and the resulting model estimated dimension assessment criteria. Our analysis hence suggests that recent results on dynamical systems showing a larger predictability horizon appear more as a result of overfitting than an intrinsic achievement of neural networks. The second question is addressed by comparing different training sets. In particular, a training set acquired from a system initially close to one of its unstable fixed points exhibits a low entropy, low complexity, behavior owing to the essentially linear local regime, before reaching a higher complexity, fully nonlinear, regime when getting closer to the attractor. This essentially allows to disentangle the contribution of the different regimes of the underlying system and to sequentially focus the learning effort on these different regimes from specific parts of the training set, effectively resulting in a curriculum. Interestingly, shortening the trajectories emanating from the fixed points and following insights based on the entropy contents lead to results consistent with the ones obtained with a larger amount of data. The last issue is related to the objective of introducing bestpractice guidelines for practitioners. The numerical test provides evidences on the effectiveness of curriculum learning, as this choice leads to consistent and accurate models even with a limited amount of data with respect to what is prescribed by ergodic theory. Moreover, the analyzed experiments have neatly shown that starting from the fixed points of the dynamical system constitutes a workaround in learning the dynamics of the system from situations where the memory is essentially harmless to the learning process. Indeed, in these memorybased models, flushing an initially incorrect memory is crucial for obtaining a relevant and faithful model able to generalize its prediction beyond the sole situations encountered during the training step. Our numerical experiments clearly show that there is an impact of the initialization of the memory on the quality of the predictions provided by the LSTM models and it is particularly severe if one learns from a set of shorttermed observations. Superior performance was consistently achieved with a memory initialized at random to avoid bias. While we here focused on the Lorenzâ€™63 system and LSTM models, we believe these findings to be widely applicable and to provide some evidencebased good practice for datadriven modeling of physical systems. Future work will focus on a finer analysis of the interplay between the structure of the dynamical system and the learning process, as well as on the choice of appropriate metrics complementing entropy levels, toward a principled learning strategy. In that sense, curriculum learning strategies can be envisioned as a valuable tool for the analysis of complex dynamical systems and open up avenues for research within the context of active learning.
Data availability statement
Data and training models used for this article will be made available on reasonable request.
References
R. Agarwal, M. Schwarzer, P.S. Castro, A.C. Courville, M. Bellemare, Deep reinforcement learning at the edge of the statistical precipice. Adv. Neural. Inf. Process. Syst. 34, 29304â€“29320 (2021)
Y.Â Bengio, J.Â Louradour, R.Â Collobert, J.Â Weston, Curriculum learning, in Proc. International Conference on Machine Learning, Montreal, Quebec, June 14â€“18 (2009)
G. Boffetta, M. Cencini, M. Falcioni, A. Vulpiani, Predictability: a way to characterize complexity. Phys. Rep. 356(6), 367â€“474 (2002)
F. Borra, A. Vulpiani, M. Cencini, Effective models and predictability of chaotic multiscale systems via machine learning. Phys. Rev. E 102(5), 052203 (2020)
S.L. Brunton, B.R. Noack, P. Koumoutsakos, Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52, 477â€“508 (2020)
F. Camastra, A. Staiano, Intrinsic dimension estimation: advances and open problems. Inf. Sci. 328, 26â€“41 (2016)
J.P. Crutchfield, B.S. McNamara, Equation of motion from a data series. Complex Syst. 1(417â€“452), 121 (1987)
M.Â deÂ Hoop, R.Â Baraniuk, J.Â Bruna, M.Â Campillo, H.Â Jasperson, S.Â Mallat, T.Â Nguyen, L.Â Seydoux, Unsupervised learning for identification of seismic signals, in Geophysical Research Abstracts, vol.Â 21 (2019)
J.P. Eckmann, D. Ruelle, Ergodic theory of chaos and strange attractors. Rev. Mod. Phys. 57, 617â€“656 (1985)
A. Eftekhari, H.L. Yap, M.B. Wakin, C.J. Rozell, Stabilizing embedology: geometrypreserving delaycoordinate maps. Phys. Rev. E 97(2), 022222 (2018)
F.Â A. Gers, D.Â Eck, J.Â Schmidhuber, Applying lstm to time series predictable through timewindow approaches, in Neural Nets WIRN Vietri01, pp. 193â€“200 (2002)
M.M. Ghazi, M. Nielsen, A. Pai, M. Modat, M.J. Cardoso, S. Ourselin, L. SĂ¸rensen, On the initialization of long shortterm memory networks. ArXiv 10454, 2019 (1912)
J.F. Gibson, J.D. Farmer, M. Casdagli, S. Eubank, An analytic approach to practical state space reconstruction. Physica D 57(1), 1â€“30 (1992)
R. Gilmore, J.M. Ginoux, T. Jones, C. Letellier, U.S. Freitas, Connecting curves for dynamical systems. J. Phys. A: Math. Theor. 43(25), 255101 (2010)
I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (MIT Press, Cambridge, 2016)
P. Grassberger, I. Procaccia, Characterization of strange attractors. Phys. Rev. Lett. 50(5), 346 (1983)
S. Hochreiter, J. Schmidhuber, Long shortterm memory. Neural Comput. 9(8), 1735â€“1780 (1997)
A. Jacot, F. Gabriel, C. Hongler, C. Neural tangent kernel: convergence and generalization in neural networks, in Advances in Neural Information Processing Systems, vol. 31, ed. by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, R. Garnett (Curran Associates Inc, Red Hook, 2018)
M. Kac, Probability and Related Topics in Physical Sciences, vol. 1 (American Mathematical Soc, Providence, 1959)
H. Kantz, T. Schreiber, Nonlinear Time Series Analysis, vol. 7 (Cambridge University Press, Cambridge, 2004)
K. Kashinath, M. Mustafa, A. Albert, J.L. Wu, C. Jiang, S. Esmaeilzadeh, K. Azizzadenesheli, R. Wang, A. Chattopadhyay, A. Singh, A. Manepalli, D. Chirila, R. Yu, R. Walters, B. White, H. Xiao, H.A. Tchelepi, P. Marcus, A. Anandkumar, P. Hassanzadeh, Prabhat, Physicsinformed machine learning: case studies for weather and climate modelling. Phil. Trans. Roy. Soc. A 379, 20200093 (2021)
N. Kuznetsov, T. Mokaev, O. Kuznetsova, E. Kudryashova, The Lorenz system: hidden boundary of practical stability and the Lyapunov dimension. Nonlinear Dyn. 102, 713â€“732 (2020)
W. La Cava, T. Helmuth, L. Spector, J.H. Moore, A probabilistic and multiobjective analysis of lexicase selection and \(\varepsilon \)lexicase selection. Evol. Comput. 27(3), 377â€“402 (2019)
E.N. Lorenz, Deterministic nonperiodic flow. J. Atmos. Sci. 20(2), 130â€“141 (1963)
S.Â Narvekar, B.Â Peng, M.Â Leonetti, J.Â Sinapov, M.Â E. Taylor, P.Â Stone, curriculum learning for reinforcement learning domains: a framework and survey. J. Mach. Learn. Res., 21(1) (2020)
G. Paladin, A. Vulpiani, Anomalous scaling laws in multifractal objects. Phys. Rep. 156(4), 147â€“225 (1987)
J. Pathak, B. Hunt, M. Girvan, Z. Lu, E. Ott, Modelfree prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach. Phys. Rev. Lett. 120(2), 024102 (2018)
J. Pathak, Z. Lu, B.R. Hunt, M. Girvan, E. Ott, Using machine learning to replicate chaotic attractors and calculate lyapunov exponents from data. Chaos Interdiscip. J. Nonlinear Sci. 27(12), 121102 (2017)
S.M. Pincus, Approximate entropy as a measure of system complexity. PNAS 88(6), 2297â€“2301 (1991)
H.Â PoincarĂ©, Les mĂ©thodes nouvelles de la mĂ©canique cĂ©leste, volumeÂ 3. GauthierVillars et fils (1899)
M. Quade, M. Abel, J. NathanÂ Kutz, S.L. Brunton, Sparse identification of nonlinear dynamics for rapid model recovery. Chaos: Interdiscip. J. Nonlinear Sci. 28(6), 063116 (2018)
M. Quade, M. Abel, K. Shafi, R.K. Niven, B.R. Noack, Prediction of dynamical systems by symbolic regression. Phys. Rev. E 94(1), 012214 (2016)
M. Raissi, P. Perdikaris, G. Karniadakis, Physicsinformed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686â€“707 (2019)
T.D. Sanger, Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE Trans. Robot. Autom. 10, 323â€“333 (1994)
T. Sauer, J.A. Yorke, M. Casdagli, Embedology. J. Stat. Phys. 65(3â€“4), 579â€“616 (1991)
M. Schmidt, H. Lipson, Distilling freeform natural laws from experimental data. Science 324(5923), 81â€“85 (2009)
P. Soviany, R.T. Ionescu, P. Rota, N. Sebe, Curriculum learning: a survey. Int. J. Comput. Vis. 130(6), 1526â€“1565 (2022)
F.Â Takens. Detecting strange attractors in turbulence. In D.Â Rand, L.S. Young, editors, Dynamical Systems and Turbulence, Warwick 1980, pp. 366â€“381, Berlin, 1981. Springer, Berlin
L.Â VanÂ der Maaten, G.Â Hinton, Visualizing data using tsne. J. Mach. Learn. Res., 9(11) (2008)
R. Varshavsky, A. Gottlieb, M. Linial, D. Horn, Novel unsupervised feature filtering of biological data. Bioinformatics 22(14), e507â€“e513 (2006)
P.R. Vlachas, J. Pathak, B.R. Hunt, T.P. Sapsis, M. Girvan, E. Ott, P. Koumoutsakos, Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Netw. 126, 191â€“217 (2020)
L.Â von Rueden, S.Â Mayer, K.Â Beckh, B.Â Georgiev, S.Â Giesselbach, R.Â Heese, B.Â Kirsch, J.Â Pfrommer, A.Â Pick, R.Â Ramamurthy, M.Â Walczak, J.Â Garcke, C.Â Bauckhage, J.Â Schuecker, Informed machine learningâ€”a taxonomy and survey of integrating knowledge into learning systems. IEEE Trans. Knowl. Data Eng. (2021). Accepted
H. Voss, M. BĂĽnner, M. Abel, Identification of continuous, spatiotemporal systems. Phys. Rev. E 57(3), 2820 (1998)
D.Â Weinshall, G.Â Cohen, D.Â Amir, Curriculum learning by transfer learning: theory and experiments with deep networks, in Proceedings of the 35th International Conference on Machine Learning, pp. 5235â€“5243. PMLR (2018)
H. Whitney, Differentiable manifolds. Ann. Math. 37(3), 645â€“680 (1936)
Y. Zhu, N. Zabaras, P.S. Koutsourelakis, P. Perdikaris, Physicsconstrained deep learning for highdimensional surrogate modeling and uncertainty quantification without labeled data. J. Comput. Phys. 394, 56â€“81 (2019)
Acknowledgements
This work was funded by the French Agence Nationale de la Recherche via the Flowcon project (ANR17ASTR0022) and the Speed project (ANR20CE23002501). L.M. gratefully acknowledges stimulating discussions with Alex Gorodetsky (University of Michigan, US). O.S. thanks Luca de Cicco (Politecnico di Bari, Italy) for exchanges on the role of entropy metrics in curriculum learning. S.C. gratefully acknowledge fruitful discussions with Angelo Vulpiani (University La Sapienza, Italy).
Author information
Authors and Affiliations
Contributions
MAB contributed to conceptualization, data curation, investigation, methodology, writing. OS contributed to conceptualization, investigation, methodology, writing. AA contributed to methodology, funding acquisition, writing (review and editing). SC contributed to methodology, funding acquisition, writing. LM contributed to investigation, methodology, funding acquisition, writing.
Corresponding author
Additional information
Quantitative AI in Complex Fluids and Complex Flows: Challenges and Benchmarks. Guest editors: Luca Biferale, Michele Buzzicotti, Massimo Cencini.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author selfarchiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Bucci, M.A., Semeraro, O., Allauzen, A. et al. Curriculum learning for datadriven modeling of dynamical systems. Eur. Phys. J. E 46, 12 (2023). https://doi.org/10.1140/epje/s10189023002698
Received:
Accepted:
Published:
DOI: https://doi.org/10.1140/epje/s10189023002698