Sparse mobile crowdsensing (MCS) is a promising paradigm for the large-scale urban sensing, which allows us to collect data from only a few areas (cell selection) and infer the data of other areas (data inference). It can significantly reduce the sensing cost while ensuring high data quality. Recently, large urban sensing systems often require multiple types of sensing data (e.g., publish two tasks on temperature and humidity respectively) to form a multi-dimensional urban sensing map. These multiple types of sensing data hold some inherent correlations, which can be leveraged to further reduce the sensing cost and improve the accuracy of the inferred results. In this paper, we study the multi-dimensional urban sensing in sparse MCS to jointly address the data inference and cell selection for multi-task scenarios. We exploit the intra-and inter-task correlations in data inference to deduce the data of the unsensed cells through the multi-task compressive sensing and then learn and select the most effective cell, task pairs by using reinforcement learning. To effectively capture the intra-and inter-task correlations in cell selection, we design a network structure with multiple branches, where branches extract the intra-task correlations for each task, respectively, and then catenates the results from all branches to capture the inter-task correlations among the multiple tasks. In addition, we present a two-stage online framework for reinforcement learning in practical use, including training and running phases. The extensive experiments have been conducted on two real-world urban sensing datasets, each with two types of sensing data, which verify the effectiveness of our proposed algorithms on multi-dimensional urban sensing and achieve better performances than the state-of-the-art mechanisms.