We use an in-house code based on polyhedral representation of particle shapes and a rigorous contact detection algorithm to investigate the evolution of orthotropic elastic response of regular polyhedral and spherical particle packings. Our DEM algorithm is able to distinguish different types of contact (face-face, face-edge, face-vertex, edge-edge, edge-vertex, vertex-vertex). A linear force law based on normal overlaps at the contact points between particles is implemented and face-face and face-edge contacts are represented by at least three and two contact points, respectively. Dense and isotropic packings of polyhedral particles with different numbers of faces were prepared by isostatic compaction. Then, they were sheared under tri-periodic boundary conditions for different values of interparticle friction coefficient. During shearing, 16 instances corresponding to a wide variety of contact orientation anisotropies were stored and relaxed a static state before applying two distinct strain probes to measure the five independent elastic moduli of the sample. By comparing the simulation data with effective medium theory (EMT), our results clearly show that the elastic moduli are functions of two microstructural parameters: 1) a constraint number that accounts for contact types (face-face and face-edge contacts between polyhedra), and 2) the contact orientation anisotropy. The proposed expression of elastic moduli isolates the direct effect of particle shape, related to the nonaffine particle displacement field from the indirect effect, related to the granular microstructure. The effect of particle shape appears at two levels: on the one hand, through four parameters in the proposed expression, which are independent of friction coefficient, and, on the other hand, through the microstructure reflected by the values of constraint number and fabric anisotropy, which depend on both particle shape and interparticle friction coefficient.