Studies in human-human interaction have introduced the concept of F-formation to describe the spatial organization of participants during social interaction. This paper aims at detecting such Fformations in images of video sequences. The proposed approach combines a voting scheme in the visual field of each participant and a memory process to make the detection in each frame robust to small, irrelevant changes of participant's behavior. Results on the MatchNMingle data set demonstrate the good performances of this approach.