Over the past decade, object detectors have demonstrated remarkable performance in various applications, such as traffic monitoring, customer tracking, and surveillance. Although advanced lightweight models have been proved to have ultra real-time speed on GPU, in edge-based video analytics system, edge servers are usually embedded devices with NPU which support general neural network operators. When we implemented deep learning models on embedded devices, images usually need to be preprocessed to the network input size. This leads to the common target detectors not being end-to-end. Image preprocessing is not the key problem of real-time inferencing on devices with high-performance CPU, but the same algorithm will bring noticeable delay on embedded devices. To overcome this, we designed YOLO-Head, a module that can handle the input of arbitrarily size according to general neural network operators. Experiment results show that YOLO-Head achieves significant (60.89%) speed improvement when 1080p image zooms to 640$$\,\times \,$$×640. Furthermore, YOLOv5-S detector with adaptive head can effectively solve the delay problem due to the image resize. The frame rate improves to 35.2 FPS, approximately 6 times faster than the convectional method in video stream processing on RK3588.