We present a new deep learning approach for short-term solar irradiance forecasting based on fisheye images. Our architecture, based on recent works on video prediction with partial differential equations, extracts spatio-temporal features modelling cloud motion to accurately anticipate future solar irradiance. Our method obtains state-of-the-art results on video prediction and 5min-ahead irradiance forecasting against strong recent baselines, highlighting the benefits of incorporating physical knowledge in deep models for real-world physical process forecasting.