The security of our recently proposed dual polarization encryption scheme of images is evaluated by numerical simulations. This consists of testing the resistance of the scheme against brute force, known-plaintext, chosen-plaintext and video sequence attacks. While some attacks are ineffective (brute force, video sequence) others are effective (known-plaintext, chosen-plaintext), but only under certain assumptions. An optimization of the setup, which is based on a regular rotation of polarization optics angles (polarizers, wave plates), is proposed associating the use of a high dynamic range for the key image, or the use of a phase-only spatial light modulator in the target and in the key image channel. The possibility of the attacker decrypting an unknown image is thus strongly reduced. The precision required for optical specifications is also evaluated, in order to ensure a good decryption for an authorized user.