A real-time architecture of medical image semantic segmentation called Fully Convolution dense Dilated Network, is proposed to improve the segmentation efficiency while ensuring high accuracy. Considering low resolution and contrast, interferences of shadows, as well as differences in nodules’ position and size, accurate ultrasound images’ segmentation cannot be obtained easily. Therefore, a novel layer that integrates the advantages of dense connectivity, dilated convolutions and factorized filters, is proposed in an attempt to remain efficient while retaining remarkable accuracy. Dense connectivity combines low-level fine segmentation with high-level coarse segmentation to extract more features from ultrasound images. Dilated convolution can expand the receptive field of the filter, and the problem of differences in nodules’ size and position can be solved with different sizes of filters. This study also introduces factorized filters into the network to further optimize the efficiency of the model. In addition, aiming at the class imbalance problem in medical image semantic segmentation, a loss function optimization method is proposed which further improves the accuracy of the network. A thorough set of experiments based on thyroid dataset show that the proposed model achieves state-of-the-art performance in terms of robustness and efficiency.