When patients operate a home infusion pump, they maybe make some mistakes, and it will be dangerous. To detect potentially life threatening errors, we design an assistance system based on observation by multiple cameras and robust spatio-temporal algorithm. Firstly, we record the video by multiple cameras when people use the infusion pump. Secondly, we use a robust MoSIFT algorithm, which detects interest points and encodes not only their local appearance but also explicitly models local motion, to describe the action. Thirdly, we recognize each individual human operating step in the use of an infusion pump to see if the patient has correctly performed the required actions in a safe sequence. The specific infusion pump used for evaluation requires 22 operation steps from 12 action classes. From the experiments show that our best classifier can obtains an average rate of 56%, and MoSIFT algorithm is robust and stable.