The understanding and modeling of inelastic scattering of thermal phonons at a solid/solid interface remain an open question. We present a fully quantum theoretical scheme to quantify the effect of anharmonic phonon-phonon scattering at an interface via nonequilibrium Green's function (NEGF) formalism. Based on the real-space scattering rate matrix, a decomposition of the interfacial spectral energy exchange is made into contributions from local and nonlocal anharmonic interactions, of which the former is shown to be predominant for high-frequency phonons whereas both are important for low-frequency phonons. The anharmonic decay of interfacial phonon modes is revealed to play a crucial role in bridging the bulk modes across the interface. The overall quantitative contribution of anharmonicity to thermal boundary conductance is found to be moderate. The present work promotes a deeper understanding of heat transport at the interface and an intuitive interpretation of anharmonic phonon NEGF formalism.