Long short-term memory networks (LSTM) achieves great success in temporal dependency modelling for chain-structured data, such as texts and speeches. An extension towards more complex data structures as encountered in 2D graphic languages is proposed in this work. Specifically, we address the problem of handwritten mathematical expression recognition, using a Tree-based BLSTM architecture allowing the direct labelling of nodes (symbol) and edges (relationship) from a graph modelling the input strokes. One major difference with the traditional approaches is that there is no explicit segmentation, recognition and layout extraction steps but a unique trainable system that produces directly a Stroke Label Graph describing a mathematical expression. The proposed system, considering no grammar, achieves competitive results in online math expression recognition domain.