Quantifying the diversity of microbial communities is a question of central importance from both a fundamental and an applied point of view. The estimation of microbial diversity has received renewed attention with the advent of large-scale metagenomic studies. Despite the remarkable sample sizes, metagenomic data sets are far from being an exhaustive census of the community on a fine taxonomic level. Hence, we must still ask what the diversity observed in a sample tells us about the diversity of the community being sampled. To answer this question, we study the link between sample composition and community structure using artificial and real communities. By evaluating the sampling properties of an unconstrained set of hypothetical community structures, we identify diversity measures that are possible to estimate and those that cannot be estimated without further assumptions. We analyze nine metagenomic data sets from a wide range of environments, and show that our findings for artificial communities are relevant for real communities. First, we argue that it is impossible to estimate the number of microbial taxa present in a community. The reason for this is simple: the sample data cannot rule out the possibility that the community contains a very large number of extremely rare taxa. We show how lack of information about rare taxa can lead to unbounded uncertainty about the number of taxa present. Second, we argue that it is also impossible to estimate the number of taxa in a relative sense. We illustrate this by applying Chao's estimator to our artificial communities: they are ranked incorrectly in the presence of a potentially large number of rare taxa. Finally, we extend our analysis to generalized diversities, including Shannon's and Simpon's, and show that the Simpson's diversity has the best estimation properties. Our claims about the impossibility of estimating the number of microbial taxa seem to contrast with previous studies. We argue that these studies rely on additional assumptions. By assuming a particular form of the taxa-abundance distribution, an estimate of the number of taxa can be obtained. However, these estimates are necessarily based on unverified extrapolation of the abundances from observed taxa to unobserved rare taxa. The possible presence of a large number of rare taxa invalidates this type of estimate. Chao's estimator is not explicitly based on a particular abundance distribution. Rather, this estimate can be interpreted as giving the lower bound of the estimates that would be obtained from a variety of distributional assumptions. As a result, Chao's estimator yields a possibly huge underestimation of the true diversity, a property that is often overlooked. However, in light of our findings, this lower bound estimate is the only robust statement that can be made about the number of taxa.