Let $S \subseteq \N$ be a numerical semigroup with multiplicity $m = \min(S \setminus \{0\})$ and conductor $c=\max(\Z \setminus S)+1$. Let $P$ be the set of primitive elements, i.e. minimal generators, of $S$, and let $L$ be the set of elements of $S$ which are smaller than $c$. Wilf's conjecture (1978) states that the inequality $|P||L| \ge c$ must hold. The conjecture has been shown to hold in case $|P| \ge m/2$ by Sammartano in 2012, and subsequently in case $|P| \ge m/3$ by the author in 2020. The main result in this paper is that Wilf's conjecture holds in case $|P| \ge m/4$ if $m$ divides $c$. The proof uses \emph{divsets} $X$, i.e. finite divisor-closed sets of monomials, as abstract models of numerical semigroups, and proceeds with estimates of the vertex-maximal matching number of the associated graph $G(X)$.