Patterns of size variation in fish are supposed to be generated by growth differences, not by egg or hatchling size variation. However, annual killifish live in temporary ponds with a limited time period available for growth and reproduction. It has therefore been hypothesized that among annual killifish, hatchling size variation should be of large relative importance to generate adaptive adult size variation. Using growth curves of 203 individuals from 18 Austrolebias species raised in a common environment, we demonstrate that hatchling size variation indeed is a main determinant of adult size variation in annual killifish, in agreement with the time constraint hypothesis. Furthermore, we find an increased early growth rate in piscivorous species augmenting their difference in size from small congeneric species. This should be adaptive if size differences determine predation success. Environmental effects of spatial location of the population of origin on hatchling size and growth suggest that the time constraint might be weakened in populations occurring near the Atlantic coast. Our study reveals how extreme environments demand specific life history solutions to achieve adaptive size variation and that there might be scope for local adaptations in growth trajectories.