In this paper, we describe an experimental test of the He-McKellar-Wilkens (HMW) topological phase with our lithium atom interferometer. The expected value of the HMW phase shift in our experiment is small and its measurement was difficult because of stray phase shifts due to small experimental defects. We start by describing our setup and we characterize the effects of the electric and magnetic fields needed to observe the HMW effect. Then, we develop a model of our interferometer signals including all the defects we have identified. After various tests of this model, we use it to suppress the largest part of the stray phase shifts. We thus obtain a series of measurements of the HMW phase: the results are 31% larger than expected and this discrepancy is probably due to some limitations of our model.