This paper uses accelerometer-embedded mobile phones to monitor one's daily physical activities for sake of changing people's sedentary lifestyle. In contrast to the previous work of recognizing user's physical activities by using a single accelerometer-embedded device and placing it in a known position or fixed orientation, this paper intends to recognize the physical activities in the natural setting where the mobile phone's position and orientation are varying, depending on the position, material and size of the hosting pocket. By specifying 6 pocket positions, this paper develops a SVM based classifier to recognize 7 common physical activities. Based on 10-folder cross validation result on a 48.2 hour data set collected from 7 subjects, our solution outperforms Yang's solution and SHPF solution by 5~6%. By introducing an orientation insensitive sensor reading dimension, we boost the overall F-score from 91.5% to 93.1%. With known pocket position, the overall F-score increases to 94.8%