Let $G=(V,E)$ be a vertex-colored graph, where $C$ is the set of colors used to color $V$. The \GM (or \sGM) problem takes as input $G$, a multiset $M$ of colors built from $C$, and asks whether there is a subset $S\subseteq V$ such that (i)~$G[S]$ is connected and (ii)~the multiset of colors obtained from $S$ equals $M$. The \CGM (or \sCGM) problem is the special case of \sGM in which $M$ is a set, and the \LGM (or \sLGM) problem is the extension of \sGM in which each vertex $v$ of $V$ may choose its color from a list $\L(v)$ of colors. We study the three problems \sGM, \sCGM, and \sLGM, parameterized by $\ell:=|V|-|M|$. In particular, for general graphs, we show that, assuming the strong exponential time hypothesis, \sCGM has no $(2-\epsilon)^\ell\cdot |V|^{O(1)}$-time algorithm, which implies that a previous algorithm, running in $O(2^\ell\cdot |E|)$ time is optimal~\cite{BBFKN11}. We also prove that \sLGM is W[1]-hard even if we restrict ourselves to lists of at most two colors. If we constrain the input graph to be a tree, then we show that \sGM can be solved in $O(4^\ell\cdot |V|)$ time but admits no polynomial-size problem kernel, while \sCGM can be solved in $O(\sqrt{2}^\ell + |V|)$ time and admits a polynomial-size problem kernel.