In this paper we describe a hardware implementation, of the MPI-2 RMA communication library primitive, devoted to a distributed Multi Processing Reconfigurable System on Chip (MP-RSoC). We designed a platform able to process communications over a custom heterogeneous MP-RSoC using our hardware MPI-2 RMA communication primitives. To implement these primitives, we have conceived a scalable Network on Chip based on a crossbar. MPI-2 RMA primitives are directly usable in hardware tasks in the MP-RSoC to transfer data between all resources, either hardware or software. We also show that using message passing for parallel programming can have benefits in term of scalability and heterogeneity. Our hardware primitives have been implemented and tested on Xilinx FPGA spartan6 board.