The pupillary light response has been shown not to be a purely reflexive mechanism but to be sensitive to higher order perceptual processes, such as covert visual attention. In the present study we examined whether the pupillary light response is modulated by stimuli that are not physically present but are maintained in visual working memory. In all conditions, displays contained both bright and dark stimuli. Participants were instructed to covertly attend and encode either the bright or the dark stimuli, which then had to be maintained in visual working memory for a subsequent change-detection task. The pupil was smaller in response to encoding bright stimuli compared to dark stimuli. However, this effect did not sustain during the maintenance phase. This was the case even when brightness was directly relevant for the working memory task. These results reveal that the encoding of task-relevant and physically present information in visual working memory is reflected in the pupil. In contrast, the pupil is not sensitive to the maintenance of task-relevant but no longer visible stimuli. One interpretation of our results is that the pupil optimizes its size for perception of stimuli during encoding; however, once stimuli are no longer visible (during maintenance), an ``optimal'' pupil size no longer serves a purpose, and the pupil may therefore cease to reflect the brightness of the memorized stimuli.