We present an anisotropic extension of the isotropic osmosis model that has been introduced by Weickert et al. [38] for visual computing applications, and we adapt it specifically to shadow removal applications. We show that in the integrable setting, linear anisotropic osmosis minimises an energy that involves a suitable quadratic form which models local directional structures. In our shadow removal applications we estimate the local structure via a modified tensor voting approach [24] and use this information within an anisotropic diffusion inpainting that resembles edge-enhancing anisotropic diffusion inpainting [39, 13]. Our numerical scheme combines the nonnegativity preserving stencil of Fehrenbach and Mirebeau [10] with an exact time stepping based on highly accurate polynomial approximations of the matrix exponential. The resulting anisotropic model is tested on several synthetic and natural images corrupted by constant shadows. We show that it outperforms isotropic osmosis, since it does not suffer from blurring artefacts at the shadow boundaries.