Understanding the flame dynamics in combustion chambers is still an important field of research. Under certain conditions large amplitude instabilities can occur, leading to degraded operating conditions or even fatal damage to the system. When considering liquid rocket engines (LRE), the power density inside the thrust chamber is so high that when these oscillations take place, they usually lead to the rapid destruction of the engine. In order to avoid this, extensive experimental testing is employed, due to the lack of reliable predictive simulation tools. However, numerical tools represent a promising, faster and cheaper alternative, but further modeling efforts and validations are required to exploit their potential. Stable, steady transcritical flames and transcritical flames modulated by external transverse acoustic waves were successfully simulated recently. However, the ability of computer codes to predict naturally unstable cases, where oscillations are self-sustained must be verified, using well controlled experiments. In this context, JAXA has developed a test bench with a single injector exhibiting large amplitude self-sustained pressure oscillations under certain operating conditions. Depending on the injection parameters and injector geometry, the transcritical flame was found to be stable or unstable. This provides a potentially rich configuration for numerical tools validation. In the present study two cases have been simulated using the AVBP-RG solver jointly developed by CERFACS and EM2C. The first case corresponds to experimentally stable injection conditions, while the second features large pressure oscillations in the chamber. Simulation results feature a higher unsteadiness, with larger fluctuations in the unstable case compared to the stable one, in good qualitative agreement with experimental data. Simulations also indicate that strong interactions take place between the exit nozzle and large-scale entropy fluctuations produced at the end of the flame. This may explain the generation of acoustic waves at this downstream boundary and this process may be at the origin of the coupling loop. In the current state of our simulations, a possible scenario for the triggering of the instability and the associated analysis are described.