Purpose: In brachytherapy, dosimetric plans are routinely calculated with the TG43 formalism which considers the patient as a simple water box. However, an accurate modeling of the physical processes considering patient heterogeneity using Monte Carlo (MC) methods is currently too time‐consuming and computationally demanding to be routinely used. As solution we implemented an accurate and fast MC simulation on graphics processing unit (GPU) for brachytherapy (HDR and LDR) applications. Methods: Based on Geant4 a MC simulation framework was developed on GPU. This framework was extended to include a hybrid GPU navigator, allowing navigation within a voxelized phantom derived from CT imaging including an analytical based structure for accurately modeling the 125I seeds (Source Tech Medical STM1251). In addition, dose scoring based on TLE including uncertainty calculations was incorporated. The implemented full GPU based modeling was compared with a classical CPU MC simulation based on GATE/GEANT4, as well as previously proposed GPU approaches based on the use of phasespace files for the seeds and their positioning based on simple replacement of voxels within the CT volumes. Results: Energy distribution from the seed and dose mapping including uncertainty were compared showing a high agreement (differences <1%). Preliminary results have shown that the GPU implementation is faster by at least two orders of magnitude compared to the GATE mono‐CPU version. A comparison between dosimetric plans based on TG43 and a full MC simulation using the GPU code for LDR prostate brachytherapy led to 40%-100% differences depending on the level of tissue heterogeneity. Conclusion: We propose a full GPU MC simulation based on Geant4, with a hybrid navigator dedicated for brachytherapy applications. Our evaluation shows large dose differences compared to the simplistic TG43 formalism in LDR and very fast execution times compatible with clinical practice.