Single object tracker based on siamese neural network have become one of the most popular frameworks in this field for its strong discrimination ability and high efficiency. However, when the task switch to multi-object tracking, the development of siamese-network based tracking methods is limited by the huge calculation cost comes from repeatedly feature extract and excessive predefined anchors. To solve these problems, we propose a siamese box adaptive multi-object tracking (SiamBAN-MOT) method with parallel detector and siamese tracker branch, which implementation in an anchor-free manner. Firstly, ResNet-50 is constructed to extract shared features of the current frame. Then, we design a siamese specific feature pyramid networks (S-FPN) to fuse the multi-scale features, which improves detection and tracking performance with the anchor-free architecture. To alleviate the duplicate feature extraction, RoI align is operated to extract all trajectories’ template feature and search region feature in a single feature map at once. After that, anchor-free based Siamese tracking network outputs the tracking result of each trajectory according to its template and search region feature. Meanwhile, current frame’s detection results are obtained from the detector for the target association. Finally, a simple novel IOU-matching scheme is designed to map the tracking results to the detection results so as to refine the tracking results and suppress the drifts caused by siamese tracking network. Through experimental verification, our method achieves competitive results on MOT17.