Person identification is an important but still challenging problem in video surveillance. This work designs a complete automatic appearance-based person identification system in online setting. The proposed system consists of three modules: background extraction and silhouette extraction; feature extraction and selection; and person identification. Grey-world normalized color features and Haralick texture features are extracted as initial feature subset. The multi-category incremental and decremental SVM (MID-SVM) algorithm is used to adaptively classify persons with the advantage of training only with few initial images and updating the needed parts. A new video database with 22 persons is created in real-life environments. The experimental results show that the proposed system succeed in person identification. In order to improve the performance of this system, a comparison of RGB and HSV colors is performed and the comparison experimental results indicate that the initial feature with HSV color and Haralick texture features works better.