Density estimation is the statistical process of constructing a probabilistic model that represents the distribution of a given dataset. By estimating this distribution, we can better understand the statistics and behavior of our data, enhancing predictions, anomaly detection, and data generation. Density estimation thus forms a crucial step in numerous data analysis and machine learning tasks. Nonetheless, the task of modeling high-dimensional distributions introduces a multitude of challenges. These primarily arise from the need to develop models that exhibit flexibility, which allows for the precise capture of complex high-dimensional patterns, and computational feasibility that becomes particularly essential during the training phase. Within this context, the thesis aims to contribute new theoretical and practical perspectives specifically designed to refine the modeling of probability density functions for high-dimensional, complex data distributions. Furthermore, we propose an innovative theory to improve the quantification of the properties of distributions, such as the thickness of their tails, under statistical assumptions relevant to the machine learning setting. The initial part of this thesis focuses on the examination of deep neural function approximators that are assured to represent probability density functions, regardless of the parameters' values. To achieve this, we introduce AFFJORD, an enhanced version of Continuous Normalizing Flows. This enhancement is made possible through augmentation, inspired by our derivation of the Jacobian of diffeomorphic transformations parameterized by Ordinary Differential Equations (ODEs). Additionally, we expand on the recent advancements in diffusion models, proposing a novel method (PSM) that enhances density estimation while accelerating training, without incurring any drawbacks in terms of inference time or memory consumption. This is achieved by exploiting the independence inherent in modeling the scores at different time points within diffusion models. The result is a flexible, rapidly optimizable, piecewise continuous normalizing flow. In the second part of the thesis, we illustrate that the finiteness of the sampling procedure from marginal distributions negatively impacts the reliability and efficiency of traditional tail estimation methods derived from Extreme Value Theory, such as the Peaks-Over-Threshold approach. To address this challenge, we devise an innovative general theory for estimating the tails of marginal distributions, particularly when there is significant variability between locations of the individual conditional distributions that underlie the marginal. Under certain regularity conditions, we demonstrate that the shape parameter of the marginal distribution corresponds to the maximum tail shape parameter of the family of conditional distributions. This estimation approach is coined as 'cross-tail estimation (CTE)'. We validate cross-tail estimation via a series of experiments conducted on both simulated and real data. Our findings showcase the improved robustness and superior quality of tail estimation in comparison to conventional methodologies and provide evidence of the correlation between overfitting and the thickness of the loss distribution tail. In conclusion, this work offers novel theoretical insights and practical perspectives, specifically tailored to enhance the modeling of probability density functions for high-dimensional, intricate data distributions. Additionally, we propose an innovative theory aimed at refining the quantification of the tail thickness of distributions, under statistical assumptions pertinent to the machine learning context. We conclude this thesis by outlining three potential avenues for future research within this field, which are then followed by our final remarks.