This paper presents an exploration of the usage of Heterogeneous Graph Attention Networks, or HGATs, for the task of Passage Retrieval. More precisely, we study how these models perform to alleviate the problem of passage contextualization, that is incorporating information about the context of a passage (its containing document, neighbouring passages, etc.) in its relevance estimation. We first propose several configurations to compute contextualized passage representations, including a document graph representation composed of contextualizing signals and judiciously modified HGAT architectures. We then present how we integrate these configurations in a neural passage ranking model. We evaluate our approach on a Passage Retrieval task on patent documents: CLEF-IP2013, as these documents possess several different contextualizing signals fully exploited in our models. Our results show that some HGAT architecture modifications allow for a better context representation leading to improved performances and stability.