This study tends to optimize and develop an innovative gypsum material by incorporating hybrid waste compounds. Using a Doehlert design, this work explores the effects of introducing paper (X1), polystyrene (X2), and polyester fibers (X3), as well as their interactions, on gypsum's thermophysical and mechanical behavior. Bulk density, thermal conductivity, flexural strength, compressive strength, and the fire resistance test were performed on the developed formulations. Statistical analysis emphasizes the significant influence of all process variables on gypsum properties. Polyester fibers notably enhance flexural strength when combined with the cooperative effect of paper and polystyrene wastes, which strike a balance between physical and mechanical characteristics. The optimum formulation parameters lead to a significant 33% decrease in bulk density, a 43% reduction in thermal conductivity, a notable 42% increase in flexural strength, and a significant reduction in compressive strength of around 50%, but still meet the 2 MPa standard. The new samples show ductile behavior, highlighting gypsum's potential as a load-bearing material with excellent thermal insulation properties and impressive fire resistance. This research significantly improves our understanding of the effectiveness of a gypsum compound incorporating waste in a hybrid manner, bringing this approach into line with sustainable development objectives.