Physics informed neural networks (PINNs) are nowadays used as efficient machine learning methods for solving differential equations. However, vanilla-PINNs fail to learn complex problems as ones involving stiff ordinary differential equations (ODEs). This is the case of some initial value problems (IVPs) when the amount of training data is too small and/or the integration interval (for the variable like the time) is too large. We propose very simple recipes to improve the training process in cases where only prior knowledge at initial time of training data is known for IVPs. For example, more physics can be easily embedded in the loss function in problems for which the total energy is conserved. A better definition of the training data loss taking into account all the initial conditions can be done. In a progressive learning approach, it is also possible to use a growing time interval with a moving grid (of collocation points) where the differential equation residual is minimized. These improvements are also shown to be efficient in PINNs modelling for solving boundary value problems (BVPs) as for the high Reynolds steady-state solution of advection-diffusion equation.