In rolling contact fatigue (RCF), failure mechanisms are known to be very sensitive to material microstructure. Yet, among the different numerical models developed to predict the RCF life, few models use a microstructure representation. A granular cohesive finite element model has been developed to simulate progressive damage of a structure subject to RCF and to investigate failure initiation mechanisms. This paper focuses on the implementation of crystal elasticity in the model. The numerical analysis of a representative volume element (RVE) validates the use of cubic elasticity to represent crystal behavior. The influence of the RVE size and the influence of boundary conditions applied on the RVE are evaluated in the finite element approximation framework. As regards the implementation of cubic elasticity in the RCF model, the generation of stress singularities at triple junctions is first highlighted. Then the average value of the intergranular shear stress is proved to be mesh size independent and therefore can be used as damage criterion. Finally, the influence of crystal elasticity on micro-cracks distribution is presented.