Computing environment is moving from computer centered designs to human-centered designs. Human’s tend to communicate wealth of information through affective states or expressions. Thus automatic analysis of user affective states have become inevitable for computer vision community. In this paper first we focus on understanding human visual system (HVS) when it decodes or recognizes facial expressions. To understand HVS, we have conducted psycho-visual experimental study with an eye-tracker, to find which facial region is perceptually more attractive or salient for a particular expression. Secondly, based on results obtained from psycho-visual experimental study we have proposed a novel framework for automatic analysis of affective states. Framework creates discriminative feature space by processing only salient facial regions to extract Pyramid Histogram of Orientation Gradients (PHOG) features. The proposed framework achieved automatic expression recognition accuracy of 95.3 % on extended Cohn-Kanade (CK+) facial expression database for six universal facial expressions. We have also discussed generalization capabilites of proposed framework on unseen data. In the last paper discusses effectiveness of proposed framework against low resolution image sequences.