The first robot taxi services are already present in certain cities of the world (\textit{e.g.}, WAYMO in San Francisco). Without surprise, most work on self-driving cars (SDC) focuses on improving the way it navigates and interacts with the environment. However, what happens in the vehicle cockpit is little explored. We currently lack tools, studies, and datasets to analyze the behaviors of the occupants of SDC in outdoor conditions and beyond driving-related activities. This paper presents a multidisciplinary, multi-sensor framework named AutoExp, to evaluate how the occupants of SDCs use their travel time, in particular non-driving actions, internal states, and the value of their time. To evaluate the proposed framework, we carried out a four-day long experiment in July 2021 using a Renault Zoe car (electric supermini urban model) programmed to behave as an SDC of SAE level 4. The experiment took place in the streets and campus of the Ecole Centrale de Nantes (ECN) in France. Using the framework, we acquired a multidisciplinary, multi-sensor dataset composed of 29 participants (18 male/11 female) conducting non-driving actions. Video recordings are 1 hour long and are associated with surveys about participants’ internal states and attitudes toward SDC. The dataset also depicts a large variety of sources of noise, both in terms of sensor data and action realization, which are common in real-world scenes but are absent in previous datasets. The proposed framework and acquired dataset will enable us to study how the occupants of SDCs use their travel time based on their postures, activities, internal states, and attitude toward this kind of vehicle. Moreover, it will enable us to develop and evaluate computer vision methods to automatically follow their in-cabin activities.