The parabolic category $\mathcal{O}$ for affine ${\mathfrak{gl}}_N$ at level $-N-e$ admits a structure of a categorical representation of $\widetilde{\mathfrak{sl}}_e$ with respect to some endofunctors $E$ and $F$. This category contains a smaller category $\mathbf{A}$ that categorifies the higher level Fock space. We prove that the functors $E$ and $F$ in the category $\mathbf{A}$ are Koszul dual to Zuckerman functors. The key point of the proof is to show that the functor $F$ for the category $\mathbf{A}$ at level $-N-e$ can be decomposed in terms of the components of the functor $F$ for the category $\mathbf{A}$ at level $-N-e-1$. To prove this, we use the following fact: a category with an action of $\widetilde{\mathfrak sl}_{e+1}$ contains a (canonically defined) subcategory with an action of $\widetilde{\mathfrak sl}_{e}$. We also prove a general statement that says that in some general situation a functor that satisfies a list of axioms is automatically Koszul dual to some sort of Zuckerman functor.