Despite numerous studies emerging on the human-horse relationship, significant gaps exist in the identification of the horse and handler factors that influence the quality of their relationship. Here, we explore key factors affecting human-animal relationships: the number of regular handlers an animal has, the length of the relationship with the handler, the number of owner changes, and the familiarity of the handler. A total of 76 horses participated in two novel object tasks (walking on novel surfaces and being touch with a novel object) to determine whether horses react differently to novel situations depending on whether they are handled by a familiar or an unfamiliar person. We observed that having multiple regular handlers negatively affected the horse reluctance towards novel surfaces and novel object. In horses used to be handled by multiple persons, 68% were showed reluctant behaviours towards the novel surfaces while 75% of the horses handled by only one person did not show reluctant behaviours. Similarly, 26% of the horses with multiple regular handlers refused to be touched with a novel object while only 13% of the horses with only one regular handler refused to be touched with the object. The relationship length between the horse and the familiar handler decreased the horse reluctance towards the novel surfaces and the novel object. The longer the relationship the less reluctant were the horses. Horses sold more than once were also more reluctant to the novel object. These horses had higher chances to refuse to be touched with the novel object than the horses still owned by their breeder or their first buyer. Finally, older horses (> 18 yo) had higher success at walking on the surface when led by someone familiar (87%) compared to led by someone unfamiliar (15%). Our findings suggest that the horse-human relationship may take time to develop as it is shaped by multiple factors involving the horse’s previous and current interactions with humans that affect their everyday life.