This paper focuses on the problem of modelling a polycrystalline ceramic material containing finite size inter-granular layers and discontinuities that occur inside the layers. The analysed composite consists of elastic grains and inter-granular layers with visco-plastic properties. The finite element method was applied to models of the material structure, i.e. the eight-noded isoparametric hexagonal finite elements that are used for grains and interfaces. This paper aims at solving the numerical simulations of the discontinuities existing in the considered material. This paper presents the influence of the initial heterogeneity of the material and different kinds of defects on the stress and displacement distribution inside the polycrystalline material subjected to uniaxial tension. The local stress concentrations can be observed in some parts of the polycrystalline structure both on the surface and inside the material. They can cause localized defect growth.