Planetary gear systems are widely used in differentials, transmissions and brake systems of many industrial machineries. Often, the components inside such a system are fully or partially submerged in oil, which serves for simultaneous lubrication and cooling. Hence, this oil recirculation is a key element for the durability and performance. The focus of this paper is two-phase flow analysis of a planetary gear system at different initial oil level and, also, its application in a transaxle brake assembly. Usually, due to the complex multi-phase flow phenomena inside the transaxle chambers, their simulation is arduous, and hence, little technical information exist about the details of their multi-phase phenomena. In order to imitate a real case situation, the numerical simulations are solved in an unsteady framework using the Realizable k-ε turbulence model together with the volume of fluid (VOF) method. Overset mesh technique is used to simulate the rotating planetary gear and the brake pad, which allows more realistic simulation of gears engagement. The aims are to predict flow characteristics, quality of lubrication and also heat transfer inside the brake during its operation. The results are presented and discussed in terms of lubrication and cooling efficiency. The outcomes can be used to improve the design of existing brake oil lubrication.