In this research we focus on the recognition of online handwritten diagrams, which is widely applied in note recording. Handwritten diagram is a kind of 2D language that spans in the plane, consisting of symbols and structures. Previous researches in the field of 2D language symbol recognition involved a complicated step such as symbol hypothesis generation and grammar description. By building a three order Markov random field on the stroke level, we not only labeled strokes but stroke relationships so that we could complete symbol grouping and symbol recognition simultaneously. The potential function in our Markov random field is log-linear model so that it is fully data-driven, and we trained it using max-margin method. We tested our method on two public handwritten diagram datasets and experiment showed that our symbol recognition method’s performance has reached state-of-the-art.