White matter pathologies such as tumors or traumatic brain injury disrupt the structure of white matter. These disruptions hamper the inference of affected pathways using tractography. A way to overcome this is to use a label fusion technique. Label fusion aims to infer the localization of the brain structure of a subject from its localization in a group of control subjects. The most common technique is known as the voting rule, where a structure is said to be present in a voxel if it's present in the majority of the voting subjects. Furthermore, this can be improved by weighting each vote by the similarity between the T1 of each voting subject and the subject to be inferred. However, these techniques only relay in the spatial localization of the structures. In this work, we introduce a way to weight the vote of each subject based on how the voted pathway is supported by the test subject's diffusion data. This is, if the diffusion data of the test subject is consistent with the direction of the voted pathway, the vote has a higher weight. We show that adding dMRI to the label fusion process achieves a similar number of true positives than the voting technique, with a 60% less of false positives. However, this incurs in a trade-off of a 40% false negatives.