This paper presents our practice with reinforcement learning (RL) in the context of a robotic art project. We adopted a design-oriented approach to RL, privileging embodied robot behaviours generated by interactive learning processes over offline evaluations of optimal agent policies. We describe aesthetic and technical issues surrounding our experiments with a spheroid robot, and share our reflections on the potential of using RL for behaviour aesthetics.