Driving behaviour understanding is of vital importance in improving transportation safety and promoting the development of Intelligent Transportation Systems (ITS). As a long-standing research topic in driving behaviour analysis, driving style identification is non-trivial. Almost all previous studies emphasize the research on the granularity of an entire trip or a driver. Inspired by the fact that an aggressive driver may drive safely at some time. In this paper, based on the widely available GPS trajectory big data that records the driving behaviours implicitly, we propose a multi-task learning (MTL) framework called semi-Traj2Graph to recognize the fine-grained driving styles in the temporal dimension accurately. The MTL framework can incorporate the learning capability of graph representation in extracting high-level and interpretable features regarding complex driving behaviours and semi-supervised in exploiting unlabelled data and reducing labelling effort. More specifically, in the graph representation learning, a multi-view graph is first built to capture a more complete view of driving behaviours from the raw GPS trajectory data, then graph convolutional neural networks (Graph-CNNs) are applied. In the semi-supervised learning, a pseudo-label labelling is adopted to make use of the unlabelled data. We evaluate the proposed framework extensively based on two taxi trajectory datasets collected from the city of Beijing and Chongqing, China, respectively. Experimental results show that semi-Traj2Graph outperforms compared to other baselines, achieving an overall accuracy of around 90%. We also implement the framework on users' smartphones via the collaborative cloud-edge computation manner to demonstrate the system usability in real cases.