We look at the behavior of structural systems under the occurrence of seismic events with the aim of identifying the fragility curves. Artificial Neural Network (ANN) empirical regression models are employed as fast-running surrogates of the (long-running) Finite Element Models (FEMs) that are typically adopted for the simulation of the system structural response. However, the use of regression models in safety critical applications raises concerns with regards to accuracy and precision. For this reason, we use the bootstrap method to quantify the uncertainty introduced by the ANN metamodel. An application is provided with respect to the evaluation of the structural damage (in this case, the maximal top displacement) of a masonry building subject to seismic risk. A family of structure fragility curves is identified, that accounts for both the (epistemic) uncertainty due to the use of ANN metamodels and the (epistemic) uncertainty due to the paucity of data available to infer the fragility parameters.