We present an analysis of dust grain emission in the diffuse interstellar medium of the Small Magellanic Cloud (SMC). This study is motivated by the availability of 170 microns ISOPHOT data covering a large part of the SMC, with a resolution enabling to disentangle the diffuse medium from the star forming regions. After data reduction and subtraction of Galactic foreground emission, we used the ISOPHOT data together with HiRes IRAS data and ATCA/Parkes combined HI column density maps to determine dust properties for the diffuse medium. We found a far infrared emissivity per hydrogen atom 30 times lower than the Solar Neighborhood value. The modeling of the spectral energy distribution of the dust, taking into account the enhanced interstellar radiation field, gives a similar conclusion for the smallest grains (PAHs and very small grains) emitting at shorter wavelength. Assuming Galactic dust composition in the SMC, this result implies a difference in the gas-to-dust ratio (GDR) 3 times larger than the difference in metallicity. This low depletion of heavy elements in dust could be specific of the diffuse ISM and not apply for the whole SMC dust if it results from efficient destruction of dust by supernovae explosions.