We report the results of a detailed study of the spectral properties of Laplace and Stokes operators, modified with a volume penalization term designed to approximate Dirichlet conditions in the limit when a penalization parameter, $\eta$, tends to zero. The eigenvalues and eigenfunctions are determined either analytically or numerically as functions of $\eta$, both in the continuous case and after applying Fourier or finite difference discretization schemes. For fixed $\eta$, we find that only the part of the spectrum corresponding to eigenvalues $\lambda \lesssim \eta^{-1}$ approaches Dirichlet boundary conditions, while the remainder of the spectrum is made of uncontrolled, spurious wall modes. The penalization error for the controlled eigenfunctions is estimated as a function of $\eta$ and $\lambda$. Surprisingly, in the Stokes case, we show that the eigenfunctions approximately satisfy, with a precision $O(\eta)$, Navier slip boundary conditions with slip length equal to $\sqrt{\eta}$. Moreover, for a given discretization, we show that there exists a value of $\eta$, corresponding to a balance between penalization and discretization errors, below which no further gain in precision is achieved. These results shed light on the behavior of volume penalization schemes when solving the Navier-Stokes equations, outline the limitations of the method, and give indications on how to choose the penalization parameter in practical cases.