The recognition of human actions in video streams is a challenging task in computer vision, with cardinal applications in e.g. brain-computer interface and surveillance. Deep learning has shown remarkable results recently, but can be found hard to use in practice, as its training requires large datasets and special purpose, energy-consuming hardware. In this work, we propose a photonic hardware approach. Our experimental setup comprises off-the-shelf components and implements an easy-to-train recurrent neural network with 16,384 nodes, scalable up to hundreds of thousands of nodes. The system, based on the reservoir computing paradigm, is trained to recognise six human actions from the KTH video database using either raw frames as inputs, or a set of features extracted with the histograms of oriented gradients algorithm. We report a classification accuracy of 91.3%, comparable to state-of-the-art digital implementations, while promising a higher processing speed in comparison to the existing hardware approaches. Because of the massively parallel processing capabilities offered by photonic architectures, we anticipate that this work will pave the way towards simply reconfigurable and energy-efficient solutions for real-time video processing.