The waveform design for beyond 5G mobile communication networks is a challenge due to asynchronous and heterogeneous scenarios. Compared to OFDM, superior spectrum confinement must be considered for future radio access technologies. Due to its characteristics, QAM-FBMC is a potential candidate. We propose a method to design a short optimal filter for QAM-FBMC system in order to reduce the total interference at the receiver. We model the total interference with and without channel effects, and describe the optimization procedure for the prototype filter design. Performance results show that the designed filters outperform QMF, NPR and Gaussian filters.