This work presents a new method for filtering the coherence map issued from Synthetic Aperture Radar (SAR) polarimetric interferometric data. For each pixel of the interferogram, an adaptive neighborhood is determined by a region growing technique driven by the amplitude image information. Then, pixels in the derived adaptive neighborhood are complex averaged to yield the filtered value of the coherence, after performing a phase compensation step. The proposed method has been applied on airborne high-resolution polarimetric interferometric SAR images. Both subjective and objective performance analysis, including coherence edge detection, shows that the proposed method provides better results than the standard phase-compensated fixed multi-look filter and a linear adaptive coherence filter proposed by Lee et al.