The objective of this paper is to provide a method of safely estimating reliability based on small samples. First, it is shown that the commonly used estimators of the parameters of the normal distribution function are biased, and they tend to lead to unconservative estimates of reliability. Then, two ways of making this estimation conservative are proposed: (1) adding constraints when a distribution is fitted to the data to bias it to be conservative, and (2) using the bootstrap method to estimate the bias needed for a given level of conservativeness. The relationship between the accuracy and the conservativeness of the estimates is explored for a normal distribution. In particular, detailed results are presented for the case when the goal is 95% likelihood to be conservative. The bootstrap approach is found to be more accurate for this level of conservativeness. It is then applied to the reliability analysis of a composite panel under thermal loading. Finally, we explore the influence of sample sizes and target probability of failure on estimates quality, and show that for a constant level of conservativeness, small samples and low probabilities can lead to a high risk of large overestimation while this risk is limited to a very reasonable value for samples above.