Point clouds are widely emerged as a promising 3D visual representation model for immersive and high quality experiences using dense point cloud. However, as they are usually made up of thousands up to billions of points, advanced techniques of data compression are essential to store and transmit this type of data. This paper improves prior work, which provided a new top-down hierarchical geometry representation based on adaptive Tree-Structured Point-Lattice Vector Quantization (TSPLVQ), to make the point cloud geometry compression more adaptive to the input point cloud characteristics. In the paper, more robust Rate-Distortion optimization process is introduced to perform efficient and accurate rate-aware splitting decisions when building and coding the tree structure. Experimental results in geometry point cloud compression, considering the tree structures and the leaves coding, show that the solution takes advantage of well-established principles that have been paramount to reach higher levels of point cloud geometry compression performance.