In the original belief function (BF) theory, a precise-valued belief structure has been widely used to represent uncertain information. However, this mentioned belief structure is difficult to effectively measure the specific hesitant situation, especially when decision makers have a set of possible values for the belief assignments of focal elements. In order to model the hesitant nature of the behavior of people to make a decision under uncertainty, we propose a hesitant fuzzy belief structure (HFBS) that is based on the BF theory and the recent hesitant fuzzy set theory. We also present the novel rule of combination of HFBS that is used and evaluated in a wearable human activity recognition (HAR) system coupled with an extreme learning machine. The evaluation of this new HFBS approach is done from two benchmark datasets. We clearly show its effectiveness and its superiority compared to various methods used classically for the wearable HAR.