Colloidal suspensions of rod-like nanoparticles are well-known to readily form liquid-crystalline phases. Using mineral nanoparticles for this purpose may impart their liquid-crystalline suspensions with original physical properties. We synthesized GdPO4 nanorods whose aqueous suspensions spontaneously organize in a nematic phase at high concentrations. The nematic phase is very well aligned by small magnetic fields, and the isotropic phase displays a very large field-induced birefringence. Moreover, the nanorods migrate to regions of high magnetic field. On the basis of magnetization measurements, we show that this unusual behavior is due to the fact that GdPO4 nanorods are actually paramagnetic. Such a paramagnetic mineral liquid crystal, easily synthesized and little sensitive to temperature, may be an interesting alternative to organometallic thermotropic liquid crystals for applications where magnetic field alignment would be more suitable than electric field alignment.