An extension of the phase field method to model interfacial damage in elastoplastic composites is proposed. In the matrix, an elastoplastic phase field is employed to model the fracture process. To introduce interfacial damage between inclusions and the matrix, a strain density function depending on the jump due to decohesion is added to the total energy. Smooth indicator functions are used to maintain the regularized character of the approximation. They weight the different terms in the energy with respect to the vicinity of interfaces. Then, the different problems (mechanical and phase field problems) are derived and an algorithmic procedure is described. Numerical examples show the capabilities of the method to handle initiation, propagation and interactions between both elastoplastic fracture and interfacial cracks in complex elastoplastic composite microstructures. It is also shown that the solutions are convergent with respect to the mesh refinement.