Multi-object tracking algorithms reach impressive performance on the benchmark datasets that they are trained and evaluated on, especially with their object detector parts tuned. When these algorithms are exposed to new videos though, the performance of the detection and tracking becomes poor, making them not usable. This paper tries to understand this behavior and discusses how we can move forward regarding these issues. Besides, we present the common errors made by the modern trackers, even when their trainable components are heavily tuned on the datasets as well as propose some directions and high-level ideas on how to proceed with those problems. We hope that it will lead to interesting discussions in the community towards solving such errors and further improving the performance of multi-object tracking.