Layered decoding is known to provide efficient and high-throughput implementation of LDPC decoders. In the practical hardware implementation of layered decoders, the performance is strongly affected by quantization. The finite precision model determines the area of the decoder, which is mainly composed of memory, especially for long frames. To be specific, in the DVB-S2,-T2 and -C2 standards, the memory can occupy up to 70% of the total area. In this paper, we focus our attention on the optimization of the number of quantization bits. Message saturation and memory size optimization are considered for the case of a DVB-S2 decoder. We show that the memory area can be reduced by 28% compared to the state-of-the-art, without performance loss.