Objective: To understand the lived experience of people with epilepsy (PWE) and their relatives, the risks associated with epilepsy, the information received from healthcare professionals, and the reaction to this information. Methods: Qualitative phenomenological study conducted between 2016 and 2018. Individual semi-directive in-depth interviews were performed based on a triangulation of sources in three study groups: PWE, relatives of PWE, and bereaved families. Interviews were analyzed continuously, using a semiopragmatic method until data saturation. Results: Interviews with PWE (N = 16), relatives of PWE (N = 8), and bereaved families (N = 10) led to several observations: (i) The stigmatizing representations of epilepsy and its constraints lead to a feeling of abnormality which determines the behavior of patients and their relatives; (ii) The global uncertainty surrounding epilepsy is an obstacle to the delivery of clear and personalized information by professionals, and, consequently, to empowerment; (iii) The communication skills of the physician have an impact on the lived experiences of patients and relatives; (iv) Better knowledge on direct mortal epilepsy-related risk could influence the perception of danger to oneself, and help find a balance between overprotection and trivialization. The experience of the patients and relatives led them to formulate concrete recommendations: (i) for the general public: to run information campaigns in order to limit stigmatization; (ii) for caregivers: to provide personalized and detailed information without minimizing the risks, in order to enable patients to “live by setting these risks”; (iii) for patients: to have a trusted person who is informed and trained in seizure management, to join patient associations. Conclusion: Our study points out that stigma, uncertainty, and lack of clarity of information are all barriers to patient empowerment. In order to provide prompt and personalized information on how to live with epilepsy while managing the risks, physicians need to develop person-centered communication skills. Future research is also required for the development of tools to facilitate this communication.