In the context of a growing popularity of Complex-Valued Neural Network (CVNN) for Polarimetric Synthetic Aperture Radar (PolSAR) applications, the input features often play a central role in classification and segmentation tasks. The socalled coherency matrix, widely used in the radar community, might limit the full potential of CVNNs. Particularly, complex-valued Pauli representation contains richer information than the coherency matrix. And the spatial coherent/local summation can also be performed by the first convolutional layers of CVNN. Letting this network learn itself the filters weights will further enhance its performance. In this paper, we propose a Complex-Valued Fully Convolutional Neural Network (CV-FCNN) which directly infers on the Pauli vector representation rather than on the coherency matrix to perform PolSAR image segmentation. The performance of CV-FCNN is then statistically evaluated on Bretigny PolSAR dataset and compared against an equivalent real-valued model.