Handwritten Mathematic Expression Recognition (HMER) algorithms with deep learning approaches have developed rapidly in recent years, most algorithms are dependent on heavy pre-training and also complex network structures. Existing architectures are build on encoder-decoder from on-line or off-line inputs to produce LATEX markup strings, or stroke-level graphs to generate symbol-level graphs. They all remain on a latent space, which is not directly related to the input data: the strokes. Using the Stroke Label Graph modelisation allows a direct connection between the input data and the output labels. In this research, we proposed a novel stroke-level graph labeling method with edge-weighted graph attention network (EGAT). This lightweight model doesn't rely on any pre-training, abandons the laborious process of encoder-decoder, is totaly end-to-end, directly accomplishes stroke-to-stroke feature extraction, and produces strokes and relations classification. Experiments show that our proposed EGAT algorithm can effectively fuse the node features as well as the weighted edge features, and predict the node and edge attributes simultaneously.