The collaboration of Internet of Things (IoT) devices promotes the computation at the network edge to satisfy latency-sensitive requests. The functionalities provided by IoT devices are encapsulated as IoT services, and the satisfaction of requests is reduced to the composition of services. Due to the hard-to-prediction of forthcoming requests, an adaptive service configuration is essential, when latency constraints are satisfied by composed services. This problem is formulated as a continuous time Markov decision process model constructed with updating system states, taking actions and assessing rewards constantly. A temporal-difference learning approach is developed to optimize the configuration, while taking long-term service latency and energy efficiency into consideration. Experimental results show that our approach outperforms the state-of-art's techniques for achieving close-to-optimal service configurations.