The results of geometrical optics modelling and optimization of the modified optical system of RATAN-600 radiotelescope are given. Suggested modifications to the optical system include the introduction of an auxiliary mirror in order to increase focal length, reduce aberrations, and expand the field of view. In this context, diffraction modelling is necessary to determine the beam pattern shape and study the polarization characteristics of the antenna. We used the aperture integration method along with the ray tracing for the calculation of the antenna beam patterns in co- and cross-polarization. Finally, the multilevel fast physical optics (PO) method was adapted for near field and beam pattern simulation of RATAN-600 radio telescope to avoid the GO approximation of the aperture field and take the finite wavelength effects into account