Aims: A growing number of PET-based methods are proposed for defining PET tumour volumes. However, their accuracies are evaluated using different phantoms and procedures, which calls for standardised evaluation. We aim to provide a 'benchmark' capable of evaluating any contouring method for a range of PET-based volumes, following recommendations by Task Group No.211 of the American Association of Medical Physics. The long-term goal is to have the benchmark publicly accessible, allowing evaluation and comparison of existing and future contouring methods. Materials and Methods: We define a benchmark by the choice of test data, truth definition, accuracy metrics, and public accessibility. The initial data include FDG PET/CT images of head-and-neck and lung cancer patients as well as numerical and experimental tumour phantoms, chosen to allow both clinical reality and the best estimate of truth. The surrogate of truth for patient images is estimated by (a) co-registered digital images of surgical specimens and (b) combination of multiple expert delineations performed on fused PET/CT scans. For phantom tumours, the absolute truth is well represented by simultaneous CT imaging (experimental) or by design (numerical). Accuracy metrics are selected to capture differences in boundary, surface and volume, for two cases where truth is (a) reliably represented by a single reference and (b) combined from multiple estimates. To promote open access to the benchmark, it is implemented as a 'plug-in' to the open-source Computational Environment for Radiotherapy Research (CERR) software. This provides for future enhancement of the benchmark as new data and methods become available. The software will also calculate the effect of the segmented volume on tumour uptake statistics and on radiotherapy dose-volume histograms. Current experiments evaluate contouring accuracy of fixed and adaptive thresholding, as well as advanced state-of-the-art algorithms. The proposed strategy for evaluating the success of the benchmark is based on the variance and validity of relative accuracy results across image sets and contouring methods. Results and Conclusions: For contour evaluation, there is no definitive image type (phantom or clinical), source of ground truth, or accuracy metric that is universally optimal. Therefore, a comprehensive benchmark calls for a combination of physical and numerical phantoms and pathology backed clinical images along with a chosen subset of accuracy metrics. The proposed design of the PET segmentation benchmark promises to provide a standard for PET contouring evaluation. Future extensions for different tracers, cancer types and organs, and support segmentation algorithms that use dynamic images are considered.