Medical devices, such as infusion pumps that deliver life-critical medication, are frequently used at home by patient's themselves as a cost saving measure. As individuals age, many are impacted by cognitive decline such that the proper sequencing of steps of a task is forgotten. Since the danger of making an error can be quite high. So, to detect errors when patients operate a home medical device, we observe them with multiple cameras and record pump sensor information. We then use a robust approach to recognize actions based on explicitly encoding motion information using MoSIFT, Which detects interest points and encodes not only their local appearance but also explicitly models local motion. Our goal is to see if the patient has correctly performed the required actions in the prescribed sequence for the device. Thus, firstly, we will evaluate how to group the requiring 22 operation steps. Secondly, we will analysis the duration of actions, and consider the user adaption. Thirdly, the order of actions is introduced, and HMM is used in our algorithm. Fourthly, as some actions are very difficult to recognize by computer vision, so the physical sensor information is borrowed in our system. Subsequently, we also consider how to fuse the results from different cameras. Finally, we also evaluate the performances when we add different numbers of videos from test subjects. From the experiments, we can see that the improvement of our system performance changes from 49% to over 80%. And if we can obtain 6-7 videos from the patients, and add them into the training dataset, the performance of our system with our simplest way can be 83.7%.