Most existing automatic kinship verification methods focus on learning the optimal distance metrics between family members. However, learning facial features and kinship features simultaneously may cause the proposed models to be too weak. In this work, we explore the possibility of bridging this gap by developing knowledge-based tensor models based on pre-trained multi-view models. We propose an effective knowledge-based tensor similarity extraction framework for automatic facial kinship verification using four pre-trained networks (i.e., VGG-Face, VGG-F, VGG-M, and VGG-S). Therefore, knowledge-based deep face and general features (such as identity, age, gender, ethnicity, expression, lighting, pose, contour, edges, corners, shape, etc.) were successfully fused by our tensor design to understand the kinship cue. Multiple effective representations are learned for kinship verification statements (children and parents) using a margin maximization learning scheme based on Tensor Cross-view Quadratic Exponential Discriminant Analysis. Through the exponential learning process, the large gap between distributions of the same family can be reduced to the maximum, while the small gap between distributions of different families is simultaneously increased. The WCCN metric successfully reduces the intra-class variability problem caused by deep features. The explanation of black-box models and the problems of ubiquitous face recognition are considered in our system. The extensive experiments on four challenging datasets show that our system performs very well compared to state-of-the-art approaches.