Electronic health records (EHRs) and medical data are classified as personal data in every privacy law, meaning that any related service that includes processing such data must come with full security, confidentiality, privacy, and accountability. Solutions for health data management, as in storing it, sharing and processing it, are emerging quickly and were significantly boosted by the COVID-19 pandemic that created a need to move things online. EHRs make a crucial part of digital identity data, and the same digital identity trends—as in self-sovereign identity powered by decentralized ledger technologies like blockchain, are being researched or implemented in contexts managing digital interactions between health facilities, patients, and health professionals. In this paper, we propose a blockchain-based solution enabling secure exchange of EHRs between different parties powered by a self-sovereign identity (SSI) wallet and decentralized identifiers. We also make use of a consortium IPFS network for off-chain storage and attribute-based encryption (ABE) to ensure data confidentiality and integrity. Through our solution, we grant users full control over their medical data and enable them to securely share it in total confidentiality over secure communication channels between user wallets using encryption. We also use DIDs for better user privacy and limit any possible correlations or identification by using pairwise DIDs. Overall, combining this set of technologies guarantees secure exchange of EHRs, secure storage, and management along with by-design features inherited from the technological stack.