Sequence-to-sequence models have lead to significant progress in keyphrase generation, butit remains unknown whether they are reli-able enough to be beneficial for document re-trieval.This study provides empirical evi-dence that such models can significantly improve retrieval performance, and introducesa new extrinsic evaluation framework that al-lows for a better understanding of the limi-tations of keyphrase generation models. Using this framework, we point out and dis-cuss the difficulties encountered with supplementing documents with –not present in text– keyphrases, and generalizing models acrossdomains. Our code is available at https://github.com/boudinfl/ir-using-kg