High herbivore abundances on trees surrounded by distantly related neighbors (phylogenetic isolation) might in part be due to local adaptation of herbivores to host trees, but this has not been tested. We studied if free-feeding and semi-concealed (shelter-building) Lepidoptera can be adapted to leaf traits of individual trees, and if this is affected by phylogenetic isolation. We performed a reciprocal transplant experiment on free-feeding and semi-concealed lepidopteran caterpillars collected from oak trees (Quercus petraea) in a mixed forest in Poland. Within a set of trees with early and a set with late budburst, we selected oak trees that varied from being surrounded by other oak trees (low phylogenetic isolation) to oaks surrounded by pine trees (high phylogenetic isolation), and collected canopy branches to obtain caterpillars. We then fed half of the caterpillars leaves from the tree they were collected from (home tree) and others on the leaves of another tree in the set (away trees) in the laboratory. We measured caterpillar mass over a five-day interval to calculate growth rate and determined aspects of leaf chemistry of each tree. Five species of Lepidoptera (Acrobasis repandana, Eudemis profundana, Operopthera brumata, Phycita roborella, Zeiraphera isertana) yielded sufficient sample sizes for statistical analyses. Overall, we found faster growth on home trees, which could be attributed to one species, E. profundana. There was no effect of phylogenetic isolation. Our results indicate that local adaptation to leaf traits of individual trees is rare in these lepidopterans, and we found no evidence that local adaptation would be more pronounced on trees that are more phylogenetically isolated from their neighbors. Therefore, the effects phylogenetic isolation on herbivory are not likely to be mediated by local adaptation to individual trees.