Minkowski sum and difference, well known as dilation and erosion in image analysis, constitute the kernel of mathematical morphology. Extending the interesting results of the later theory from 2D images to 3D meshes is very promising. However, this requires the development of robust algorithms for the computation of Minkowski operations, which is far from being an easy task. The contributing vertices concept has been introduced for the computation of Minkowski sum of convex polyhedra, and later extended for the computation of non-convex/convex pairs of polyhedra. For Minkowski difference, the available literature is poor. In this work, we demonstrate the duality of the contributing vertices concept w.r.t. Minkowski operations, and propose an exact and efficient Contributing Vertices-based Minkowski Difference (CVMD) algorithm for polyhedra. Our algorithm operates on convex polyhedra and on pairs of convex/non-convex polyhedra without any modification. We also show its beneficial application where the second operand is represented in 3D by an implicit surface or a point cloud, among other possible representations. The conducted benchmarks show that CVMD largely outperforms an indirect Nef polyhedrabased approach we implemented in order to validate our results. All our implementations produce exact results and supplementary materials are provided.