In this paper, we present a reinforcement learning approach for multi-agent communication in order to learn what to communicate, when and to whom. This method is based on introspective agents that can reason about their own actions and data so as to construct appropriate communicative acts. We propose an extension of classical reinforcement learning algorithms for multi-agent communication. We show how communicative acts and memory can help solving non-markovity and a synchronism issues in MAS.