This paper deals with an inverse problem applied to the field of building physics to experimentally estimate three sorption isotherm coefficients of a wood fiber material. First, the mathematical model, based on convective transport of moisture, the Optimal Experiment Design (OED) and the experimental set-up are presented. Then measurements of relative humidity within the material are carried out, after searching the OED, which is based on the computation of the sensitivity functions and a priori values of the unknown parameters employed in the mathematical model. The OED enables to plan the experimental conditions in terms of sensor positioning and boundary conditions out of 20 possible designs, ensuring the best accuracy for the identification method and, thus, for the estimated parameter. Two experimental procedures were identified: i) single step of relative humidity from 10% to 75% and ii) multiple steps of relative humidity 10-75-33-75% with an 8-day duration period for each step. For both experiment designs, it has been shown that the sensor has to be placed near the impermeable boundary. After the measurements, the parameter estimation problem is solved using an interior point algorithm to minimize the cost function. Several tests are performed for the definition of the cost function, by using the L^2 or L^\infty norm and considering the experiments separately or at the same time. It has been found out that the residual between the experimental data and the numerical model is minimized when considering the discrete Euclidean norm and both experiments separately. It means that two parameters are estimated using one experiment while the third parameter is determined with the other experiment. Two cost functions are defined and minimized for this approach. Moreover, the algorithm requires less than 100 computations of the direct model to obtain the solution. In addition, the OED sensitivity functions enable to capture an approximation of the probability distribution function of the estimated parameters. The determined sorption isotherm coefficients calibrate the numerical model to fit better the experimental data. However, some discrepancies still appear since the model does not take into account the hysteresis effects on the sorption capacity. Therefore, the model is improved proposing a second differential equation for the sorption capacity to take into account the hysteresis between the main adsorption and desorption curves. The OED approach is also illustrated for the estimation of five of the coefficients involved in the hysteresis model. To conclude, the prediction of the model with hysteresis are compared with the experimental observations to illustrate the improvement of the prediction.